Repolode
Submit
unclecode

unclecode/crawl4ai

Open-source Python web crawler that turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines.

View on GitHubWebsite
Stars
85,717
Forks
8,908
Open issues
261
Last commit
3 days ago

What is it?

Crawl4AI is an open-source web crawler and scraper written in Python. It fetches pages with a real browser and converts them into clean Markdown that language models can read, which makes it useful for retrieval-augmented generation, AI agents and data pipelines. It can also extract structured JSON from pages, either with CSS and XPath schemas that need no LLM, or with any LLM provider supported through LiteLLM.

The library handles JavaScript-heavy pages, infinite scroll, sessions, cookies, proxies and persistent browser profiles. It supports deep crawls with breadth-first, depth-first and best-first strategies, plus URL discovery from sitemaps and Common Crawl. A Docker server exposes REST endpoints such as /md, /crawl and /screenshot, and a crwl command-line tool covers quick jobs.

The project also offers Crawl4AI Cloud, a hosted API with search, SERP and answer endpoints, plus an MCP server for connecting AI agents. The core library and self-hosted server are free to run yourself under Apache-2.0.

Key features

  • 01Converts pages to clean Markdown with filters that strip menus, footers and boilerplate.
  • 02Extracts structured JSON via CSS/XPath schemas or any LLM provider.
  • 03Runs JavaScript, waits for elements and scrolls full pages for lazy-loaded content.
  • 04Supports deep crawls (BFS, DFS, best-first) and bulk URL crawling.
  • 05Offers browser control with persistent profiles, proxies, sessions and stealth options.
  • 06Ships a Docker REST server, a crwl CLI and an MCP endpoint.

Use cases

  • Feeding web content into a RAG pipeline or vector database.
  • Giving an AI agent the ability to read and summarize live web pages.
  • Scraping product or pricing pages into structured data without writing parsers by hand.
  • Archiving documentation sites as Markdown for offline search.

Topics

#web-crawler#web-scraping#llm#rag#markdown#playwright#ai-agents#data-extraction#ai#crawler

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects