Repolode
Submit
apify

apify/crawlee-python

A Python library for building reliable web crawlers and scrapers, with HTTP and headless browser modes, proxy rotation and persistent storage.

Open-source alternative to:Scrapy

View on GitHubWebsite
Stars
9,651
Forks
823
Open issues
93
Last commit
10 hours ago

What is it?

Crawlee is a Python library for web scraping and browser automation. It handles the plumbing of crawling: discovering links, queuing URLs, running requests in parallel, retrying failed pages and storing extracted data in machine-readable formats, so developers can focus on the extraction logic.

The library offers a unified interface for plain HTTP crawling and headless browser crawling. Its BeautifulSoupCrawler parses HTML with BeautifulSoup, while PlaywrightCrawler drives a browser through Playwright for pages that rely on client-side JavaScript. Request routing sends URLs to different handlers, and built-in proxy rotation and session management are provided.

Crawlee can also download files such as HTML, PDF, and images. Crawlers are ordinary Python scripts with type hints, so they can be embedded in other applications. A project scaffold can be created with the Crawlee CLI using prepared templates.

Key features

  • 01Unified interface for HTTP and headless browser crawling.
  • 02Automatic parallel crawling based on available system resources.
  • 03Automatic retries and integrated proxy rotation with session management.
  • 04Persistent URL queue and pluggable storage for tabular data and files.
  • 05Request routing that sends URLs to specific handlers.

Use cases

  • Extracting structured product or article data from websites into datasets.
  • Collecting web content, such as HTML or PDFs, to feed AI, LLM, or RAG pipelines.
  • Automating scraping of JavaScript-heavy sites with Playwright.
  • Running resumable, long-lived crawls that survive interruptions.

Quick start

  1. 1

    Install Crawlee with all features using pip

    python -m pip install 'crawlee[all]'
  2. 2

    Install the Playwright browser dependencies

    playwright install
  3. 3

    Verify the installation by printing the version

    python -c 'import crawlee; print(crawlee.__version__)'

Things to know

  • Full browser crawling requires installing Playwright browser dependencies separately.

Topics

#web-scraping#web-crawler#browser-automation#playwright#beautifulsoup#asyncio#proxy#apify#automation#crawler

FAQ

What is crawlee-python?
A Python library for building reliable web crawlers and scrapers, with HTTP and headless browser modes, proxy rotation and persistent storage. Crawlee is a Python library for web scraping and browser automation. It handles the plumbing of crawling: discovering links, queuing URLs, running requests in parallel, retrying failed pages and storing extracted data in machine-readable formats, so developers can focus on the extraction logic.
Is crawlee-python free to use? What license does it have?
crawlee-python is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is crawlee-python still maintained?
Actively maintained. The last commit to crawlee-python was on Nov 6, 2026.
What can I use crawlee-python for?
Extracting structured product or article data from websites into datasets. Collecting web content, such as HTML or PDFs, to feed AI, LLM, or RAG pipelines. Automating scraping of JavaScript-heavy sites with Playwright. Running resumable, long-lived crawls that survive interruptions.
How do I install crawlee-python?
Install Crawlee with all features using pip: python -m pip install 'crawlee[all]' Install the Playwright browser dependencies: playwright install Verify the installation by printing the version: python -c 'import crawlee; print(crawlee.__version__)'
Where can I get crawlee-python?
The source code and installation instructions are on GitHub: https://github.com/apify/crawlee-python

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects