Devtoolwatch BoardCrawl4AI review
Crawl4AI is an open-source Python web crawler and scraper that turns websites into clean, LLM-ready Markdown or structured JSON for RAG and agents.
TL;DR
Crawl4AI is an open-source Python crawler that converts web pages into clean Markdown and structured JSON for RAG, AI agents and data pipelines. You can run it yourself as a library, CLI or Docker server, or use the hosted Crawl4AI Cloud with one key. The cloud adds web search, SERP data and answers, plus MCP access for coding agents.
- 86.7kGitHub stars
- +504new stars, week of Oct 25
- 358.1kdownloads a week, +10% vs 4-week avg
- #14in RAG, data and memory this week
Crawl4AI at a glance
- Category
- RAG, data and memory
- Maintained by
- UncleCode (individual, @unclecode on GitHub)
- License
- Apache 2.0
- Language
- Python, JavaScript, Shell
- Repository
- github.com/unclecode/crawl4ai
- Website
- crawl4ai.com
- PyPI package
- crawl4ai 0.9.5
- Latest release
- v0.9.5 on GitHub, Oct 14, 2026
- First published
- May 9, 2024
What Crawl4AI does
- Crawl4AI converts pages into Markdown with headings, lists, tables, code blocks and a numbered citation list of links.
- Crawl4AI filters menus, footers and boilerplate with pruning, BM25 query-based or LLM content filters to produce fit Markdown.
- Crawl4AI extracts structured data with CSS, XPath or regex schemas without an LLM, or with any LLM provider into typed JSON.
- Crawl4AI deep-crawls sites with BFS, DFS or best-first strategies and can resume long crawls after a crash.
- Crawl4AI handles dynamic pages by running JavaScript, waiting for elements and scrolling the full page for infinite scroll.
- Crawl4AI controls browsers with persistent profiles, sessions, proxies, stealth mode and remote CDP connections across Chromium, Firefox and WebKit.
- Crawl4AI Cloud adds web search, Google SERP data and question answering endpoints, plus an MCP server for agents.
How it works
Crawl4AI runs as a Python library using an async browser pool, as the crwl command line tool, or as a Docker server exposing a REST API and MCP. Self-hosted browsers run on your machine with your own proxies. Crawl4AI Cloud runs browsers for you behind endpoints such as /scrape, /extract and /search, and connects to Claude Code, Codex, Cursor and OpenCode over MCP.
How to install and start using Crawl4AI
With pip
pip install -U crawl4aiSet up the browser
crawl4ai-setupCheck the installation
crawl4ai-doctorRequirements
- Python 3.10 or later
- A browser installed via crawl4ai-setup or Playwright for local crawling
- A CRAWL4AI_API_TOKEN for the self-hosted Docker server
- An LLM provider key for LLM extraction or CLI questions when self-hosting
First steps
- Install Crawl4AI with pip, then run crawl4ai-setup once to install the browser.
- Create an AsyncWebCrawler, call arun with a URL, and print result.markdown.
- From the terminal, run crwl with a URL and -o markdown to get a page as Markdown.
- For the cloud, verify your email to get a key and call the /scrape endpoint with it.
Who Crawl4AI is for
- Developers building RAG pipelines who need web pages as clean Markdown
- AI agent builders who want web scraping and search through MCP
- Teams extracting structured JSON from sites, with or without an LLM
- Python developers who want full control over browsers, sessions and proxies
Limitations to know
- Crawl4AI's web search, SERP and answer endpoints exist only in the cloud, not in the library or self-hosted server.
- Crawl4AI self-hosting leaves JS-heavy pages and bot walls to your own settings and proxies.
- Crawl4AI's SERP endpoint is in beta and the answer endpoint is experimental.
- Crawl4AI's free cloud tier allows 1,000 credits a month, 10 requests a minute, 2 at a time.
Is Crawl4AI growing? Usage data
In the week of Oct 25–31, 2026, Crawl4AI ranked #132 on the Devtoolwatch Board and #14 in RAG, data and memory, with a momentum score of 56 out of 100. Its main package had 358.1k downloads that week, +10% against its own 4-week average.
Show the weekly numbers
| Week of | New stars |
|---|---|
| Aug 9, 2026 | 895 |
| Aug 16, 2026 | 922 |
| Aug 23, 2026 | 846 |
| Aug 30, 2026 | 1.7k |
| Sep 6, 2026 | 1.1k |
| Sep 13, 2026 | 1.1k |
| Sep 20, 2026 | 480 |
| Sep 27, 2026 | 448 |
| Oct 4, 2026 | 506 |
| Oct 11, 2026 | 631 |
| Oct 18, 2026 | 501 |
| Oct 25, 2026 | 504 |
On Hacker News
- Crawl4AI is an open-source, LLM-friendly web crawler and scraper, 7 points and 2 comments (Apr 29, 2025)
- Crawl4AI: Open-Source Web Crawler for Seamless AI Data Scraping, 6 points and 0 comments (Sep 29, 2024)
The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.
Crawl4AI alternatives
Other RAG, data and memory tools on the Board in the week of Oct 25–31, 2026, by momentum:
| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | XERJXERJ is a local-first search engine and MCP server for AI agents, packaged as one Rust binary with an Elasticsearch-compatible API. | 17.9k | – |
| #2 | MemvidMemvid is a single-file memory layer for AI agents that packages data, embeddings, search indexes, and metadata into one portable .mv2 file. | 17.3k | – |
| #3 | LEANNLEANN is a Python vector database and RAG library that recomputes embeddings on demand to keep indexes small on personal devices. | 13.7k | – |
| #4 | GraphifyGraphify is an AI coding assistant skill that turns a codebase, with its docs, PDFs, images and videos, into a queryable knowledge graph. | 140k | 480.7k |
| #5 | MarkItDownMarkItDown is a lightweight Python utility from Microsoft that converts PDFs, Office documents, images, audio and other files to… | 193.2k | 4.7M |
| #6 | FirecrawlFirecrawl is an API and open source project to search, scrape, crawl, and interact with web pages, returning LLM-ready markdown or JSON… | 197.4k | 2.9M |
Recent Crawl4AI releases
Crawl4AI FAQ
Is Crawl4AI free and open source?
Yes. Crawl4AI's source code is on GitHub (unclecode/crawl4ai) under the Apache 2.0 license, so you can use it for free. Its website also links to a pricing page (https://crawl4ai.com/#pricing) for any paid plans.
How do I install Crawl4AI?
Run pip install -U crawl4ai. The crawl4ai package needs Python 3.10 or newer. The install section above has the other options from the project's README.
Is Crawl4AI still maintained?
Yes. The unclecode/crawl4ai repository was last updated on Oct 21, 2026, and the latest release, v0.9.5 on GitHub, came out on Oct 14, 2026.
How popular is Crawl4AI?
Crawl4AI has 86.7k GitHub stars as of Nov 1, 2026. It gained 504 new stars in the week of Oct 25–31, 2026, against a 12-week average of 812 a week. Its main package had 358.1k downloads that week, +10% against its own 4-week average. That made it #14 among RAG, data and memory tools on the Devtoolwatch Board.
Crawl4AI vs Firecrawl: how do they compare?
Crawl4AI and Firecrawl are both tracked by Devtoolwatch as RAG, data and memory tools. Crawl4AI has 86.7k GitHub stars and Firecrawl has 197.4k. In the week of Oct 25–31, 2026, Crawl4AI's main package had 358.1k downloads and Firecrawl's had 2.9M. Firecrawl is an API and open source project to search, scrape, crawl, and interact with web pages, returning LLM-ready markdown or JSON for AI agents. Firecrawl review.
Crawl4AI vs Browser Use: how do they compare?
Crawl4AI and Browser Use are both tracked by Devtoolwatch. Crawl4AI has 86.7k GitHub stars and Browser Use has 119.7k. In the week of Oct 25–31, 2026, Crawl4AI's main package had 358.1k downloads and Browser Use's had 1.5M. Browser Use review.
What are the best alternatives to Crawl4AI?
Other RAG, data and memory tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: XERJ (#1), Memvid (#2), LEANN (#3), Graphify (#4), MarkItDown (#5). The alternatives table above has their numbers.
Is Crawl4AI free?
Crawl4AI is free to run yourself as a library or Docker server. Crawl4AI Cloud gives 1,000 free credits every month, with 10 requests a minute and 2 at a time. Paid plans start from $10 a month with 120 a minute and 10 at a time.
Is Crawl4AI open source?
Crawl4AI is open source under the Apache-2.0 license. The project says the self-hosted version stays open source.
What is Crawl4AI used for?
Crawl4AI is used to turn websites into clean Markdown or structured JSON for RAG, AI agents and data pipelines. It also deep-crawls sites, takes screenshots and PDFs, and collects media and links.
Does Crawl4AI use Playwright?
Crawl4AI's README gives a Playwright command to install Chromium by hand if browser setup fails, and lists playwright among its topics. It supports Chromium, Firefox and WebKit.
How does Crawl4AI work?
Crawl4AI drives a browser through an async browser pool, loads pages, optionally runs JavaScript, and converts the content into Markdown or extracts JSON with schemas or an LLM. Caching skips repeated fetches. The cloud version runs browsers for you.
How do I use Crawl4AI in n8n?
Crawl4AI Cloud exposes an MCP server over streamable HTTP at https://api.crawl4ai.com/mcp. The homepage lists n8n among clients that can connect with an Authorization Bearer header.
Sources
- unclecode/crawl4ai on GitHub
- Crawl4AI README
- crawl4ai on PyPI
- Crawl4AI website
- Release v0.9.5 (Oct 14, 2026)
- Release v0.9.4 (Sep 23, 2026)
- Release v0.9.3 (Aug 31, 2026)
- Hacker News: Crawl4AI is an open-source, LLM-friendly web crawler and scraper
- Hacker News: Crawl4AI: Open-Source Web Crawler for Seamless AI Data Scraping
- Devtoolwatch Board, the week of Oct 25–31, 2026
Crawl4AI README badge
Maintainers can show the tool's weekly Board rank in their README:
[](https://index.devtoolwatch.com/tools/crawl4ai/)How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].