Repolode
Submit
ScrapeGraphAI

ScrapeGraphAI/Scrapegraph-ai

Python library that uses LLMs and graph-based pipelines to extract structured data from websites and local documents based on a natural-language prompt.

Open-source alternative to:Firecrawl

View on GitHubWebsite
Stars
31,929
Forks
3,367
Open issues
22
Last commit
13 days ago

What is it?

ScrapeGraphAI is a Python web scraping library that replaces hand-written selectors with a natural-language prompt. You describe the information you want, and the library runs a pipeline that fetches the page or local file (XML, HTML, JSON, Markdown and similar) and uses a large language model to return the extracted data as a dictionary.

The library provides several prebuilt pipelines. SmartScraperGraph handles a single page, SearchGraph works across the top results of a search engine, and SmartScraperMultiGraph and ScriptCreatorMultiGraph cover multiple sources. Other pipelines generate audio output or reusable Python scripts. Multi-source variants can run LLM calls in parallel.

Models can be OpenAI, Groq, Azure, Gemini or MiniMax through their APIs, or local models served by Ollama. Browser rendering uses Playwright, which must be installed separately. A managed cloud API and SDKs are offered by the same company as a separate, paid option.

Key features

  • 01Extracts data from a page using a plain-language prompt instead of CSS or XPath selectors.
  • 02Supports websites and local files such as XML, HTML, JSON and Markdown.
  • 03Offers single-page, multi-page and search-results pipelines.
  • 04Works with OpenAI, Groq, Azure, Gemini and local models via Ollama.
  • 05Can generate audio output or Python scripts from scraped content.

Use cases

  • Pulling company details, founders or contact links from many websites without writing parsers.
  • Collecting structured data from pages whose layout changes often.
  • Feeding web content into LLM or RAG pipelines in a structured form.
  • Running private extraction with a local model through Ollama.

Topics

#web-scraping#llm#data-extraction#python#web-crawler#rag#ollama#ai-crawler#ai-scraping#ai-search

FAQ

What is Scrapegraph-ai?
Python library that uses LLMs and graph-based pipelines to extract structured data from websites and local documents based on a natural-language prompt. ScrapeGraphAI is a Python web scraping library that replaces hand-written selectors with a natural-language prompt. You describe the information you want, and the library runs a pipeline that fetches the page or local file (XML, HTML, JSON, Markdown and similar) and uses a large language model to return the extracted data as a dictionary.
Is Scrapegraph-ai free to use? What license does it have?
Scrapegraph-ai is open source under the MIT license. Check the license terms before using it in commercial products.
Is Scrapegraph-ai still maintained?
Actively maintained. The last commit to Scrapegraph-ai was on Oct 6, 2026.
What can I use Scrapegraph-ai for?
Pulling company details, founders or contact links from many websites without writing parsers. Collecting structured data from pages whose layout changes often. Feeding web content into LLM or RAG pipelines in a structured form. Running private extraction with a local model through Ollama.
Where can I get Scrapegraph-ai?
The source code and installation instructions are on GitHub: https://github.com/ScrapeGraphAI/Scrapegraph-ai

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects