Repolode
Submit
YOLOW92

YOLOW92/pagesift

Rust web crawler and extraction engine that turns websites, PDFs and Office files into clean Markdown and structured data for AI agents and RAG pipelines.

Open-source alternative to:FirecrawlJina Reader

View on GitHub
Stars
245
Forks
32
Open issues
0
Last commit
yesterday

What is it?

PageSift is a web scraping and content extraction engine written in Rust, aimed at feeding AI agents and LLM pipelines. It fetches a URL or uploaded file and converts the main content into Markdown or structured data. It covers HTML pages, PDF, Word, Excel, PowerPoint, EPUB and CSV, and can crawl whole sites with adaptive politeness and per-host rate limits. Headless browser rendering is used only when a page needs JavaScript to show its content.

The project ships as a single binary with a CLI, a Docker server, and Python and Node.js packages. Its HTTP API is compatible with Firecrawl v1/v2 and provides a Jina Reader-style `/r/` endpoint, so existing Firecrawl client code can point at a local instance. It also runs as a shared MCP server for agents, and can log each fetch to an audit file.

The engine is designed with bounded memory: queues are capped, and it reads cgroup memory limits to size its concurrency. Focused crawls can be guided by a query, scoring links by relevance to it. Robots.txt compliance and SSRF protections for private IP ranges are included. The project is licensed under AGPL-3.0, with a commercial license available.

Key features

  • 01Converts HTML, PDF, Word, Excel, PowerPoint, EPUB and CSV into Markdown or structured data.
  • 02Offers Firecrawl v1/v2-compatible APIs and a Jina Reader-style endpoint.
  • 03Ships Python and Node.js SDKs plus a CLI and Docker image.
  • 04Renders JavaScript pages with a headless browser only when needed.
  • 05Runs as a shared MCP server with an audit log of fetches.
  • 06Keeps memory bounded by capping queues and reading cgroup limits.

Use cases

  • Building a RAG knowledge base from websites and internal documents.
  • Giving LLM agents a tool to fetch clean page content.
  • Replacing a hosted Firecrawl or Jina Reader dependency with a self-hosted service.
  • Batch-converting PDFs and Office files to Markdown for indexing.

Things to know

  • Licensed under AGPL-3.0, a copyleft license that requires sharing source code when the software is used over a network.
  • Some benchmarks show it behind rivals, such as spider on the A2 replay and fast crawl timings.
  • The 200M-request throughput figures come from a synthetic local origin with no network, so they are not real-internet speeds.

In collections

Browser automation for AI agentsOpen-source AI agent frameworks & appsMCP serversStatic site generators & documentation tools

Topics

#web-scraping#web-crawler#rag#markdown#pdf-extraction#ai-agents#rust#mcp#browser-automation#data-extraction

FAQ

What is pagesift?
Rust web crawler and extraction engine that turns websites, PDFs and Office files into clean Markdown and structured data for AI agents and RAG pipelines. PageSift is a web scraping and content extraction engine written in Rust, aimed at feeding AI agents and LLM pipelines. It fetches a URL or uploaded file and converts the main content into Markdown or structured data. It covers HTML pages, PDF, Word, Excel, PowerPoint, EPUB and CSV, and can crawl whole sites with adaptive politeness and per-host rate limits. Headless browser rendering is used only when a page needs JavaScript to show its content.
Is pagesift free to use? What license does it have?
pagesift is open source under the AGPL-3.0 license. Check the license terms before using it in commercial products.
Is pagesift still maintained?
Actively maintained. The last commit to pagesift was on Nov 6, 2026.
What can I use pagesift for?
Building a RAG knowledge base from websites and internal documents. Giving LLM agents a tool to fetch clean page content. Replacing a hosted Firecrawl or Jina Reader dependency with a self-hosted service. Batch-converting PDFs and Office files to Markdown for indexing.
Where can I get pagesift?
The source code and installation instructions are on GitHub: https://github.com/YOLOW92/pagesift

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects