Repolode
Submit
alirezamika

alirezamika/autoscraper

A Python web scraper that learns extraction rules from sample data on a page and reuses them on similar or new pages.

View on GitHub
Stars
8,074
Forks
831
Open issues
2
Last commit
3 months ago

What is it?

AutoScraper is a Python library for extracting data from web pages without hand-writing selectors. You give it a URL or HTML content plus a list of sample values you want, such as text, links, or values from specific HTML tags. It learns the rules that locate those elements and returns similar items from the page.

Once built, the scraper object can be saved to disk and loaded later. It can then pull similar content from other pages with the same layout, or retrieve the exact same elements from new URLs in the order the samples were given. Requests options such as proxies and custom headers can be passed through to the underlying requests calls.

The library targets Python 3 and is installable from PyPI or directly from the Git repository.

Key features

  • 01Learns scraping rules from a few sample values instead of manual selectors.
  • 02Extracts text, URLs, or specific HTML tag values.
  • 03Returns similar elements from other pages with the same structure.
  • 04Saves and loads trained scraper models.
  • 05Passes custom requests parameters such as proxies and headers.

Use cases

  • Pulling titles or prices from a list of pages that share one layout.
  • Turning a website into a simple API by wrapping a saved scraper in a web framework.
  • Quickly prototyping a scraper without writing CSS or XPath selectors.

Quick start

  1. 1

    Install the library from PyPI

    pip install autoscraper
  2. 2

    Import AutoScraper, build a scraper from a URL and sample data, then fetch similar results

Things to know

  • Learned rules can break when a page's HTML structure changes.
  • The README does not document JavaScript rendering, so dynamically loaded content may not be captured.

Topics

#web-scraping#python#scraper#automation#crawler#data-extraction#ai#artificial-intelligence#machine-learning#scrape

FAQ

What is autoscraper?
A Python web scraper that learns extraction rules from sample data on a page and reuses them on similar or new pages. AutoScraper is a Python library for extracting data from web pages without hand-writing selectors. You give it a URL or HTML content plus a list of sample values you want, such as text, links, or values from specific HTML tags. It learns the rules that locate those elements and returns similar items from the page.
Is autoscraper free to use? What license does it have?
autoscraper is open source under the MIT license. Check the license terms before using it in commercial products.
Is autoscraper still maintained?
Updated within 6 months. The last commit to autoscraper was on Jul 29, 2026.
What can I use autoscraper for?
Pulling titles or prices from a list of pages that share one layout. Turning a website into a simple API by wrapping a saved scraper in a web framework. Quickly prototyping a scraper without writing CSS or XPath selectors.
How do I install autoscraper?
Install the library from PyPI: pip install autoscraper Import AutoScraper, build a scraper from a URL and sample data, then fetch similar results
Where can I get autoscraper?
The source code and installation instructions are on GitHub: https://github.com/alirezamika/autoscraper

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects