# Crawl4AI review

> Crawl4AI is an open-source Python web crawler and scraper that turns websites into clean, LLM-ready Markdown or structured JSON for RAG and agents.

Devtoolwatch Board, the week of Oct 25–31, 2026. Canonical page: https://index.devtoolwatch.com/tools/crawl4ai/

## TL;DR

Crawl4AI is an open-source Python crawler that converts web pages into clean Markdown and structured JSON for RAG, AI agents and data pipelines. You can run it yourself as a library, CLI or Docker server, or use the hosted Crawl4AI Cloud with one key. The cloud adds web search, SERP data and answers, plus MCP access for coding agents.

## Crawl4AI at a glance

- **Category:** [RAG, data and memory](https://index.devtoolwatch.com/categories/rag-data-and-memory/)
- **Maintained by:** UncleCode (individual, [@unclecode](https://github.com/unclecode) on GitHub)
- **License:** Apache 2.0
- **Language:** Python, JavaScript, Shell
- **Repository:** [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
- **Website:** [crawl4ai.com](https://crawl4ai.com)
- **PyPI package:** [crawl4ai](https://pypi.org/project/crawl4ai/) 0.9.5
- **Latest release:** [v0.9.5](https://github.com/unclecode/crawl4ai/releases/tag/v0.9.5) on GitHub, Oct 14, 2026
- **First published:** May 9, 2024

## What Crawl4AI does

- Crawl4AI converts pages into Markdown with headings, lists, tables, code blocks and a numbered citation list of links.
- Crawl4AI filters menus, footers and boilerplate with pruning, BM25 query-based or LLM content filters to produce fit Markdown.
- Crawl4AI extracts structured data with CSS, XPath or regex schemas without an LLM, or with any LLM provider into typed JSON.
- Crawl4AI deep-crawls sites with BFS, DFS or best-first strategies and can resume long crawls after a crash.
- Crawl4AI handles dynamic pages by running JavaScript, waiting for elements and scrolling the full page for infinite scroll.
- Crawl4AI controls browsers with persistent profiles, sessions, proxies, stealth mode and remote CDP connections across Chromium, Firefox and WebKit.
- Crawl4AI Cloud adds web search, Google SERP data and question answering endpoints, plus an MCP server for agents.

### How it works

Crawl4AI runs as a Python library using an async browser pool, as the crwl command line tool, or as a Docker server exposing a REST API and MCP. Self-hosted browsers run on your machine with your own proxies. Crawl4AI Cloud runs browsers for you behind endpoints such as /scrape, /extract and /search, and connects to Claude Code, Codex, Cursor and OpenCode over MCP.

## How to install Crawl4AI

**With pip**

```
pip install -U crawl4ai
```

**Set up the browser**

```
crawl4ai-setup
```

**Check the installation**

```
crawl4ai-doctor
```

### Requirements

- Python 3.10 or later
- A browser installed via crawl4ai-setup or Playwright for local crawling
- A CRAWL4AI_API_TOKEN for the self-hosted Docker server
- An LLM provider key for LLM extraction or CLI questions when self-hosting

### First steps

1. Install Crawl4AI with pip, then run crawl4ai-setup once to install the browser.
2. Create an AsyncWebCrawler, call arun with a URL, and print result.markdown.
3. From the terminal, run crwl with a URL and -o markdown to get a page as Markdown.
4. For the cloud, verify your email to get a key and call the /scrape endpoint with it.

## Who Crawl4AI is for

- Developers building RAG pipelines who need web pages as clean Markdown
- AI agent builders who want web scraping and search through MCP
- Teams extracting structured JSON from sites, with or without an LLM
- Python developers who want full control over browsers, sessions and proxies

### Limitations

- Crawl4AI's web search, SERP and answer endpoints exist only in the cloud, not in the library or self-hosted server.
- Crawl4AI self-hosting leaves JS-heavy pages and bot walls to your own settings and proxies.
- Crawl4AI's SERP endpoint is in beta and the answer endpoint is experimental.
- Crawl4AI's free cloud tier allows 1,000 credits a month, 10 requests a minute, 2 at a time.

## Usage data (the week of Oct 25–31, 2026)

- Board rank: #132 overall, #14 in RAG, data and memory, momentum score 56
- Downloads that week: 358.1k (+10% vs 4-week average)
- New GitHub stars per week (last 12 weeks): Aug 9: 895; Aug 16: 922; Aug 23: 846; Aug 30: 1.7k; Sep 6: 1.1k; Sep 13: 1.1k; Sep 20: 480; Sep 27: 448; Oct 4: 506; Oct 11: 631; Oct 18: 501; Oct 25: 504

## Crawl4AI alternatives

| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | [XERJ](https://index.devtoolwatch.com/tools/xerj-org-xerj/) | 17.9k | – |
| #2 | [Memvid](https://index.devtoolwatch.com/tools/memvid-memvid/) | 17.3k | – |
| #3 | [LEANN](https://index.devtoolwatch.com/tools/startrail-org-leann/) | 13.7k | – |
| #4 | [Graphify](https://index.devtoolwatch.com/tools/graphify-labs-graphify/) | 140k | 480.7k |
| #5 | [MarkItDown](https://index.devtoolwatch.com/tools/markitdown/) | 193.2k | 4.7M |
| #6 | [Firecrawl](https://index.devtoolwatch.com/tools/firecrawl/) | 197.4k | 2.9M |

## Crawl4AI FAQ

### Is Crawl4AI free and open source?

Yes. Crawl4AI's source code is on GitHub (unclecode/crawl4ai) under the Apache 2.0 license, so you can use it for free. Its website also links to a pricing page (https://crawl4ai.com/#pricing) for any paid plans.

### How do I install Crawl4AI?

Run `pip install -U crawl4ai`. The crawl4ai package needs Python 3.10 or newer. The install section above has the other options from the project's README.

### Is Crawl4AI still maintained?

Yes. The unclecode/crawl4ai repository was last updated on Oct 21, 2026, and the latest release, v0.9.5 on GitHub, came out on Oct 14, 2026.

### How popular is Crawl4AI?

Crawl4AI has 86.7k GitHub stars as of Nov 1, 2026. It gained 504 new stars in the week of Oct 25–31, 2026, against a 12-week average of 812 a week. Its main package had 358.1k downloads that week, +10% against its own 4-week average. That made it #14 among RAG, data and memory tools on the Devtoolwatch Board.

### Crawl4AI vs Firecrawl: how do they compare?

Crawl4AI and Firecrawl are both tracked by Devtoolwatch as RAG, data and memory tools. Crawl4AI has 86.7k GitHub stars and Firecrawl has 197.4k. In the week of Oct 25–31, 2026, Crawl4AI's main package had 358.1k downloads and Firecrawl's had 2.9M. Firecrawl is an API and open source project to search, scrape, crawl, and interact with web pages, returning LLM-ready markdown or JSON for AI agents. [Firecrawl review](https://index.devtoolwatch.com/tools/firecrawl/)

### Crawl4AI vs Browser Use: how do they compare?

Crawl4AI and Browser Use are both tracked by Devtoolwatch. Crawl4AI has 86.7k GitHub stars and Browser Use has 119.7k. In the week of Oct 25–31, 2026, Crawl4AI's main package had 358.1k downloads and Browser Use's had 1.5M. [Browser Use review](https://index.devtoolwatch.com/tools/browser-use/)

### What are the best alternatives to Crawl4AI?

Other RAG, data and memory tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: XERJ (#1), Memvid (#2), LEANN (#3), Graphify (#4), MarkItDown (#5). The alternatives table above has their numbers.

### Is Crawl4AI free?

Crawl4AI is free to run yourself as a library or Docker server. Crawl4AI Cloud gives 1,000 free credits every month, with 10 requests a minute and 2 at a time. Paid plans start from $10 a month with 120 a minute and 10 at a time.

### Is Crawl4AI open source?

Crawl4AI is open source under the Apache-2.0 license. The project says the self-hosted version stays open source.

### What is Crawl4AI used for?

Crawl4AI is used to turn websites into clean Markdown or structured JSON for RAG, AI agents and data pipelines. It also deep-crawls sites, takes screenshots and PDFs, and collects media and links.

### Does Crawl4AI use Playwright?

Crawl4AI's README gives a Playwright command to install Chromium by hand if browser setup fails, and lists playwright among its topics. It supports Chromium, Firefox and WebKit.

### How does Crawl4AI work?

Crawl4AI drives a browser through an async browser pool, loads pages, optionally runs JavaScript, and converts the content into Markdown or extracts JSON with schemas or an LLM. Caching skips repeated fetches. The cloud version runs browsers for you.

### How do I use Crawl4AI in n8n?

Crawl4AI Cloud exposes an MCP server over streamable HTTP at https://api.crawl4ai.com/mcp. The homepage lists n8n among clients that can connect with an Authorization Bearer header.

## Sources

1. [unclecode/crawl4ai on GitHub](https://github.com/unclecode/crawl4ai)
2. [Crawl4AI README](https://github.com/unclecode/crawl4ai/blob/main/README.md)
3. [crawl4ai on PyPI](https://pypi.org/project/crawl4ai/)
4. [Crawl4AI website](https://crawl4ai.com)
5. [Release v0.9.5 (Oct 14, 2026)](https://github.com/unclecode/crawl4ai/releases/tag/v0.9.5)
6. [Release v0.9.4 (Sep 23, 2026)](https://github.com/unclecode/crawl4ai/releases/tag/v0.9.4)
7. [Release v0.9.3 (Aug 31, 2026)](https://github.com/unclecode/crawl4ai/releases/tag/v0.9.3)
8. [Hacker News: Crawl4AI is an open-source, LLM-friendly web crawler and scraper](https://news.ycombinator.com/item?id=43827993)
9. [Hacker News: Crawl4AI: Open-Source Web Crawler for Seamless AI Data Scraping](https://news.ycombinator.com/item?id=41689534)
10. [Devtoolwatch Board, the week of Oct 25–31, 2026](https://index.devtoolwatch.com/)