Open-source Python web crawler that turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines.
85.7K8.9KPythonApache-2.0Updated 3 days ago
#web-crawler
Open-source projects tagged “web-crawler”
Curated GitHub repositories related to web-crawler, with AI summaries and key stats.
Sort:Most stars
2 repositoriesClear filters
Python tool that audits any website from its homepage URL, crawling it against 52 SEO rules and producing PDF, XLSX and Markdown reports.
654105PythonMITUpdated 27 days ago