crawl4ai
Open-source crawler built for LLMs/agents — "any website → LLM-ready markdown"
An open-source crawler built for LLMs and agents: "any website → LLM-ready markdown".
Search engines, scrapers, and data infrastructure for AI pipelines.
Key features
- Turns any website into clean, LLM-ready Markdown
- Structured extraction via CSS/XPath schemas or any LLM into typed JSON
- Browser control: sessions, proxies, cookies, stealth mode, user scripts
- Async browser pool and caching; content filters remove boilerplate
Good for
- RAG pipelines needing clean markdown from websites
- AI agents gathering web data with full browser control
Stack & tags
PythonDockerweb-scraperweb-crawlerllm-datamarkdownragstructured-extraction
Quick start
pip install -U crawl4ai
crawl4ai-setupFrom the project README on GitHub.
Screenshots
Screenshots from the project’s GitHub README.