Repolode
Submit
D4Vinci

D4Vinci/Scrapling

Python web scraping framework covering single HTTP requests, stealth browsers, parsing, and full concurrent crawls with exports.

View on GitHubWebsite
Stars
88,646
Forks
9,146
Open issues
9
Last commit
3 hours ago

What is it?

Scrapling is a Python web scraping framework that combines fast HTTP requests, stealth browsers, a parser with CSS, XPath, and text selection, and a concurrent spider engine in one library. It can fetch static pages, render JavaScript, interact with sites, and capture the API calls behind them.

For larger jobs, its Scrapy-like Spider API supports per-domain throttling, proxy rotation, retries, pause and resume via a checkpoint directory, streaming output, and JSON, JSONL, CSV, and XML export. Built-in templates cover sitemaps, feeds, link following, and Shopify stores.

Scrapling can also be driven from the terminal through its CLI and interactive shell, or by AI agents through an MCP server and an installable Agent Skill. It can convert pages to Markdown for use in RAG pipelines.

Key features

  • 01Fetchers for plain HTTP requests, stealth browsers, and persistent sessions with proxy rotation.
  • 02Concurrent spiders with per-domain throttling, AutoThrottle, robots.txt support, and retries.
  • 03Pause and resume long crawls using a checkpoint directory.
  • 04CSS, XPath, text, regex, and similar-element selection with adaptive element tracking.
  • 05Export results to JSON, JSONL, CSV, or XML.
  • 06MCP server, Agent Skill, and CLI for use from AI agents and the terminal.

Use cases

  • Extracting a few fields from a static or JavaScript-rendered page in a Python script.
  • Running a large crawl of a website with pause, resume, and structured exports.
  • Letting an AI agent browse pages and extract data through MCP.
  • Converting web pages into Markdown corpora for retrieval-augmented generation.

Quick start

  1. 1

    Read the install instructions in the README

  2. 2

    Fetch a single page with the Fetcher example from the README

Topics

#web-scraping#crawler#python#headless-browser#data-extraction#mcp#parsing#proxy#ai#ai-scraping

FAQ

What is Scrapling?
Python web scraping framework covering single HTTP requests, stealth browsers, parsing, and full concurrent crawls with exports. Scrapling is a Python web scraping framework that combines fast HTTP requests, stealth browsers, a parser with CSS, XPath, and text selection, and a concurrent spider engine in one library. It can fetch static pages, render JavaScript, interact with sites, and capture the API calls behind them.
Is Scrapling free to use? What license does it have?
Scrapling is open source under the BSD-3-Clause license. Check the license terms before using it in commercial products.
Is Scrapling still maintained?
Actively maintained. The last commit to Scrapling was on Oct 19, 2026.
What can I use Scrapling for?
Extracting a few fields from a static or JavaScript-rendered page in a Python script. Running a large crawl of a website with pause, resume, and structured exports. Letting an AI agent browse pages and extract data through MCP. Converting web pages into Markdown corpora for retrieval-augmented generation.
How do I install Scrapling?
Read the install instructions in the README Fetch a single page with the Fetcher example from the README
Where can I get Scrapling?
The source code and installation instructions are on GitHub: https://github.com/D4Vinci/Scrapling

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects