Open-source projects tagged “crawler”

Open-source Python web crawler that turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines.

Open-source web data platform that turns any website into a structured API via scraping, crawling, search, monitoring and AI extraction.

Chrome extension that detects which anti-bot systems, CAPTCHAs and fingerprinting techniques a website uses, with confidence scores.