Firecrawl
Turn the web into LLM-ready data — AI crawler/search API, self-hostable
"Turn the entire web into LLM-ready data": an AI crawler/search API whose open-source edition is self-hostable.
Search engines, scrapers, and data infrastructure for AI pipelines.
Key features
- Web data API: search, scrape, crawl, map and batch scrape
- LLM-ready output: markdown, structured JSON, screenshots
- Handles proxies, rate limits and JS-heavy pages automatically
- Interact with pages via AI prompts or code before extracting
Good for
- AI agents needing reliable web data for retrieval
- Developers turning any URL into clean markdown or JSON
Stack & tags
PythonJavaScriptweb-scrapingweb-crawlerllm-datamarkdown-extractionagent-toolsopen-source
Quick start
from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
result = app.scrape('firecrawl.dev')From the project README on GitHub.
Screenshots
Screenshots from the project’s GitHub README.
Similar open-source projects
yt-dlp
★ 194,511The video downloader — essential for creators and researchers, top-tier star count
awesome-llm-apps
★ 140,347100+ open-source AI agent/RAG app collection — a selection reference library