Repolode
Submit
spider-rs

spider-rs/spider

A concurrency-first web crawler and scraping engine in Rust that renders JavaScript only where needed.

Open-source alternative to:Scrapy CloudApify

View on GitHubWebsite
Stars
2,790
Forks
316
Open issues
4
Last commit
6 hours ago

What is it?

Spider is a web crawling and scraping engine written in Rust. It streams pages as they are fetched, uses plain HTTP by default, and launches headless Chrome only for pages that require JavaScript rendering. The same API works for a single local script or a distributed worker setup.

The library handles link discovery, depth and concurrency limits, delays, robots.txt handling, subdomain control, and user-agent configuration. Proxies, retries, rate limiting, and stealth options are built in. Users can also run the same engine on Spider Cloud, switching to managed proxies and anti-bot handling with a configuration change.

Bindings and tools are available for Rust, the command line, Node.js, Python, and as an MCP server for AI assistants.

Key features

  • 01Streams pages as they are fetched instead of waiting for the crawl to finish.
  • 02Uses HTTP by default and launches headless Chrome only for JavaScript-heavy pages.
  • 03Controls concurrency, crawl depth, request delay, and robots.txt compliance.
  • 04Offers built-in proxy, retry, rate limiting, and stealth options.
  • 05Provides bindings for Rust, Node.js, and Python, plus a CLI and MCP server.

Use cases

  • Feeding web content into vector stores for LLM or RAG pipelines.
  • Monitoring sites for SEO changes or price updates.
  • Exporting crawled pages as Markdown, JSON, or WARC files.
  • Driving headless Chrome for AI browsing agents.

Quick start

  1. 1

    Add the Rust library to your project

    cargo add spider
  2. 2

    Or install the command-line tool

    cargo install spider_cli
  3. 3

    Write a crawler using Website::new and crawl().await (see README example)

Things to know

  • Full Spider Cloud features require a paid or free API key from a third-party service.

Topics

#web-crawler#web-scraping#rust#headless-chrome#scraping#automation#web-data#ai-agent#crawler#spider

FAQ

What is spider?
A concurrency-first web crawler and scraping engine in Rust that renders JavaScript only where needed. Spider is a web crawling and scraping engine written in Rust. It streams pages as they are fetched, uses plain HTTP by default, and launches headless Chrome only for pages that require JavaScript rendering. The same API works for a single local script or a distributed worker setup.
Is spider free to use? What license does it have?
spider is open source under the MIT license. Check the license terms before using it in commercial products.
Is spider still maintained?
Actively maintained. The last commit to spider was on Oct 20, 2026.
What can I use spider for?
Feeding web content into vector stores for LLM or RAG pipelines. Monitoring sites for SEO changes or price updates. Exporting crawled pages as Markdown, JSON, or WARC files. Driving headless Chrome for AI browsing agents.
How do I install spider?
Add the Rust library to your project: cargo add spider Or install the command-line tool: cargo install spider_cli Write a crawler using Website::new and crawl().await (see README example)
Where can I get spider?
The source code and installation instructions are on GitHub: https://github.com/spider-rs/spider

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects