How to Use Crawl4AI: The Beginner's Guide to LLM-Friendly Web Scraping

What Is Crawl4AI, Exactly?

Crawl4AI is an open-source Python library that turns messy web pages into clean, LLM-ready Markdown: you give it a URL, it drives a real browser, strips the chrome, and hands you text your model can actually reason over. Learning how to use Crawl4AI is really learning one pipeline — pages in, clean Markdown out — that you then plug into RAG systems, AI agents, and dataset builds. It is Apache-2.0 licensed, free to self-host forever, and the project is large (about 86,700 stars on GitHub) because it solved the exact problem every agent builder hits: raw HTML is garbage for LLMs.

Say the mental model once and the rest of this crawl4ai tutorial follows: Crawl4AI = web pages in, clean Markdown out, purpose-built for LLMs — not a general-purpose scraper with AI bolted on, but a crawler whose output format IS the product. This guide takes you from zero to a working crawl on the current line (v0.9.5, released October 2026): install, your first crawl, the markdown lesson, extraction, dynamic pages, batch crawling, the Docker server, the CLI, the MCP hookup, and the beginner mistakes to skip. A quick note on scope: the hosted cloud tier exists (1,000 free credits a month), but this guide is the self-hosted open-source lane, which is the lane the project is famous for. If containers are new to you, start with our Docker beginner guide — the Docker-server section below will make much more sense after it.

Who is it for? Anyone feeding the web to a model: RAG builders who need clean source documents, agent developers whose bots must read live pages, and data folks building training or evaluation sets. You need Python with pip and a machine that can run a headless browser — no API key required until you touch LLM extraction.

What Do You Need Before You Start?

A working Python install with pip, enough disk space for a browser (~a few hundred megabytes), and internet access — that is the whole shopping list for the Python path. The browser is installed by Crawl4AI's own setup step, not by you hunting down drivers, and LLM-powered extraction is the only part that asks for an API key or a local model.

One responsible-scraping rule, said once: only crawl pages you are allowed to crawl. Check a site's robots.txt and terms of service before you point an automated browser at it, and do not hammer small sites with aggressive parallel crawling — politeness is part of the tool working well, not just ethics.

How Do You Install Crawl4AI?

Two commands, and the second one only ever runs once: pip install -U crawl4ai gets the library, then crawl4ai-setup installs the browser Crawl4AI drives. Say it once: crawl4ai-setup is a one-time install of the browser — never put it in a loop, a retry script, or a container entrypoint that runs it every boot. The crawl4ai install is complete after that, and crawl4ai-doctor will check the whole installation for you if something feels off later.

pip install -U crawl4ai
crawl4ai-setup
crawl4ai-doctor

If crawl4ai-doctor reports everything green, you are ready for your first crawl. On a fresh machine this takes a few minutes on the first run; every later run skips the browser download entirely.

How Do You Run Your First Crawl in Python?

The first crawl is five lines: open an AsyncWebCrawler, await an arun() on a URL, and read result.markdown. This is the canonical crawl4ai python quickstart from the official docs, and it is the shape every later example in this guide extends.

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print(result.markdown[:300])  # Print first 300 chars

if __name__ == "__main__":
    asyncio.run(main())

That result.markdown is the direct HTML-to-Markdown conversion — the whole point of the library, right there. Two knobs matter from day one: BrowserConfig(headless=True) controls the browser, and CrawlerRunConfig(cache_mode=CacheMode.BYPASS) controls the run — the default cache mode is BYPASS, which means you get fresh content every time, not a stale cached copy.

What Is the Difference Between Raw and Fit Markdown?

Raw markdown is everything the page contains; fit markdown is everything the page contains after Crawl4AI's content filter throws away the navigation, footers, ads, and boilerplate — and fit markdown is what you actually want to feed a model. The markdown lesson, said once: raw is the conversion, fit is the conversion after the filter, and only fit is LLM-ready out of the box.

You get the filter by passing a markdown generator with a content filter, like this:

from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
from crawl4ai.content_filter_strategy import PruningContentFilterLXML

md_generator = DefaultMarkdownGenerator(
    content_filter=PruningContentFilterLXML(threshold=0.4, threshold_type="fixed")
)

config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    markdown_generator=md_generator,
)

Then result.markdown.raw_markdown gives you the unfiltered conversion and result.markdown.fit_markdown gives you the same content after the filter — compare the two once on a real article page and you will never skip the filter again. The filter adds roughly 50 milliseconds, and if you set no generator at all you will see only raw Markdown, which is the single most common "why is my output noisy" beginner trap.

How Do You Extract Structured Data With Crawl4AI?

When you need fields instead of prose — prices, titles, specs — you reach for an extraction strategy, and Crawl4AI gives you two families: schema-based strategies that need no LLM, and LLM strategies for when the page structure fights you. The crawl4ai extraction family, at a glance:

Start with the no-LLM family whenever the page is structured: define the schema once, reuse it forever, and pay nothing per page. JsonCssExtractionStrategy.generate_schema(html, llm_config=...) can even draft a reusable schema from sample HTML when you do not want to hand-write selectors. When the page is unstructured or hostile, LLMExtractionStrategy with LLMConfig(provider="openai/gpt-4o", api_token=...) — or provider="ollama/llama3.3" against a local model so you pay nothing at all — reads the markdown and returns your schema filled in.

Extraction results land in result.extracted_content as JSON, and one more trick for the toolbox: the raw:// prefix lets you feed raw HTML straight in instead of a URL, which is perfect for testing a strategy on saved page source. If local models are new to you, our Ollama beginner guide gets one running on your machine — it pairs beautifully with free LLM extraction.

How Do You Crawl JavaScript-Heavy Pages?

Enable JavaScript in the browser config and Crawl4AI renders the page before extracting, so single-page apps and lazy-loaded content stop being invisible. BrowserConfig(headless=True, java_script_enabled=True) is the switch, and wait_for plus js_code handle the two classic dynamic-page problems: content that arrives late, and content that only appears after an interaction.

For multi-step flows, session_id reuses the same browser page across crawls, js_only=True runs JavaScript without re-navigating, and kill_session() cleans up when you are done — say the dynamic-pages lesson once: render first with JS enabled, wait for what you need, interact with js_code, and kill the session when the flow ends. If you ever need full scripted browser automation beyond scraping — logins, form wizards, screenshots on a schedule — our Playwright beginner guide covers that lane; Crawl4AI stays in the crawl-and-extract lane.

How Do You Crawl Many Pages at Once?

Hand a URL list to arun_many() with a MemoryAdaptiveDispatcher and Crawl4AI crawls in parallel while respecting your machine's memory — this is the difference between a toy script and a real pipeline. Batch mode processes the whole list; stream=True yields results as each page finishes, which is what you want when you are feeding a live RAG ingest queue. For sites where you do not know the URL list up front, AdaptiveCrawler follows links intelligently instead of you hand-building the list.

Watch two things at scale: memory (the adaptive dispatcher exists because browsers are heavy — a few hundred megabytes per page adds up), and politeness (parallel crawling multiplies your request rate, which is where the robots.txt rule from the top of this guide earns its keep).

How Do You Run Crawl4AI as a Docker Server?

One docker run command turns Crawl4AI into an HTTP API your whole team — including non-Python stacks — can call, and this is the Docker-server rule, said once: Python teams use the library directly, everyone else goes through the server or the CLI. This is the crawl4ai docker path at its simplest: one container, one token, the full engine behind HTTP. The container listens on port 11235, and every endpoint requires an API token:

docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g \
  -e CRAWL4AI_API_TOKEN=$TOKEN unclecode/crawl4ai:latest

Test it with the /md endpoint, which returns a page as Markdown — the same clean output the Python path gives you:

curl -s http://localhost:11235/md \
  -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
  -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown

The REST surface covers the full workflow — here is the endpoint map worth bookmarking:

A dashboard lives at /dashboard and an interactive playground at /playground on the same port — open the playground once and you will stop guessing at config options. Running this on a server instead of your laptop? Our VPS hosting guide covers picking a box with enough RAM for a browser fleet.

How Does the crwl CLI Work?

crwl is the command-line face of the same engine: one URL and an output flag gives you Markdown in your terminal, with deep crawling and LLM questions as flags. Three invocations cover 90 percent of CLI use:

crwl https://news.ycombinator.com -o markdown
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
crwl https://www.example.com/products -q "Extract all product prices"

The third form needs an LLM key, configured once with crwl config — it sends the page's markdown to a model and returns your question answered from the page. Think of the CLI as the server's keyboard shortcut: no Python, no HTTP client, just answers in the terminal.

How Do You Connect Crawl4AI to Your AI Agent?

Crawl4AI ships a Model Context Protocol server, so agents like Claude Code can browse the web through your crawler instead of their built-in fetch. The crawl4ai mcp hookup against the cloud tier is one command: claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp --header "Authorization: Bearer $CRAWL4AI_KEY" — after that, the agent calls your crawler as a tool. The self-hosted Docker image also exposes MCP for connecting agents to your own server, keeping every fetched page inside your infrastructure.

If MCP itself is new, read our what is MCP explainer first — the short version is that MCP is the open standard for giving agents tools, and Crawl4AI is one of the more useful tools an agent can have. This is the "web scraping for llm" endgame: the agent asks, your crawler fetches clean Markdown, the agent reasons over it, and no HTML ever touches the context window — Crawl4AI's whole reputation is being the llm friendly web scraper for exactly this loop.

Crawl4AI vs Firecrawl: Which Should You Pick?

Draw the lane line once and be done with it: Crawl4AI is the open-source lane — Apache-2.0, Python, free forever, you run the browser and you own the infrastructure. Firecrawl is the paid-API lane — you send URLs to their service and they run the browsers for you. Same job (JS rendering, Markdown, JSON extraction), different trade: control and zero marginal cost versus zero infrastructure and a bill.

Crawl4AI does offer a middle ground if you want the API shape without the infrastructure: the hosted cloud tier gives 1,000 free credits a month (10 requests a minute, 2 concurrent), with paid plans from $10 a month unlocking 120 requests a minute and 10 concurrent crawls, across /scrape, /extract, /search, /serp, /answer, and batch endpoints. Pick self-hosted when you crawl at volume or care where the pages get fetched; pick managed when you want to ship today. They are different lanes, never rivals — and this guide is the self-hosted one.

What Are the Beginner Mistakes to Avoid?

Every beginner hits the same wall, and it is always one of these:

Work through them in order the first time something breaks and you will solve most failures in minutes. The pattern underneath all six is the same: Crawl4AI is explicit about its moving parts — browser, cache, filter, key, token — and every failure is just one of them misconfigured.

Sources

• Crawl4AI Documentation — Quickstart (v0.9.x) — https://docs.crawl4ai.com/core/quickstart/
• GitHub — unclecode/crawl4ai (README: install, cloud tiers, star/fork counts, v0.9.5 release) — https://github.com/unclecode/crawl4ai
• OpenSourceAlternatives — Crawl4AI item (Crawl4AI vs Firecrawl comparison) — https://www.opensourcealternatives.to/item/crawl4ai

Frequently Asked Questions

Is Crawl4AI really free?

Yes. Crawl4AI is Apache-2.0 open source and free to self-host forever — install it with pip and crawl as much as your own hardware allows, with no seat or page limits. The optional hosted cloud tier starts at a free 1,000 credits a month, with paid plans from $10 a month for higher rate limits and concurrency.

Do I need Python to use Crawl4AI?

No. The Python library is the deepest integration, but the Docker server exposes the same engine as REST endpoints (/md, /html, /crawl, and more), and the crwl CLI gives you Markdown, deep crawls, and LLM questions from any terminal. Use the library for Python pipelines and the server or CLI for everything else.

What is the difference between Crawl4AI and Firecrawl?

Crawl4AI is the open-source lane: you self-host it free forever and run the browser yourself (Apache-2.0, Python). Firecrawl is the paid-API lane: you send URLs to their service and they run the browsers for you. Both do JS rendering, Markdown output, and JSON extraction; pick the lane that matches your budget and infrastructure appetite.

Why is my Crawl4AI output full of menus and footers?

You are reading raw_markdown without a content filter. Pass a DefaultMarkdownGenerator with PruningContentFilterLXML to your CrawlerRunConfig and read result.markdown.fit_markdown instead — that is the filtered, LLM-ready version. The filter costs about 50 milliseconds per page.

Do I need an OpenAI key for LLM extraction?

Only for LLMExtractionStrategy. The CSS, XPath, and regex strategies extract structured data with no key and no per-page cost. If you want LLM extraction without paying, point LLMConfig at provider="ollama/llama3.3" and run the model locally.