Repolode
Submit
itsOwen

itsOwen/CyberScraper-2077

An LLM-powered web scraper, built on Streamlit, that extracts data from pages using OpenAI, Gemini, Ollama or LiteLLM models.

Open-source alternative to:Browse AIOctoparse

View on GitHub
Stars
3,293
Forks
382
Open issues
1
Last commit
8 days ago

What is it?

CyberScraper 2077 is a web scraper that uses large language models to turn web pages into structured data. You enter a URL and describe in natural language what you want, and the tool fetches the page with Playwright and asks the chosen model to pull out the matching content. Supported backends include OpenAI, Gemini, local models via Ollama, and any model exposed through a LiteLLM proxy.

The project ships a Streamlit web interface and can be run locally with Python 3.10+ or inside Docker. Results can be exported as JSON, CSV, HTML, SQL or Excel, and CSV output can be uploaded to Google Sheets. Multi-page scraping accepts URL patterns and page ranges, and the project also includes Tor routing for .onion addresses and optional captcha handling via a browser window.

The author notes that open-source local models need a capable machine and may require prompt tuning, and recommends OpenAI or Gemini for better instruction following.

Key features

  • 01Extracts data from web pages using natural-language instructions via OpenAI, Gemini or Ollama models.
  • 02Supports LiteLLM proxies to use any model they expose.
  • 03Exports results to JSON, CSV, HTML, SQL or Excel.
  • 04Uploads CSV data to Google Sheets.
  • 05Scrapes multiple pages using URL patterns and page ranges.
  • 06Routes .onion URLs through the Tor network.

Use cases

  • Pulling product or listing data from paginated websites into a spreadsheet.
  • Prototyping structured extraction from pages without writing per-site selectors.
  • Running scraping with a local Ollama model to keep data on your own machine.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/itsOwen/CyberScraper-2077.git
  2. 2

    Install the required packages

    pip install -r requirements.txt
  3. 3

    Install Playwright

    playwright install
  4. 4

    Start the Streamlit app and open http://localhost:8501

    streamlit run main.py

Things to know

  • Requires Python 3.10 or higher.
  • Needs an OpenAI or Gemini API key, or a local Ollama setup for open-source models.
  • The author recommends OpenAI and Gemini over open-source LLMs, which may be slower and need a capable system.
  • Captcha bypass and current-browser features do not work on Docker, and Windows users are directed to Docker.

Topics

#web-scraping#llm#openai#gemini-api#ollama#streamlit#playwright#data-extraction#ai-scraping#llm-scraper

FAQ

What is CyberScraper-2077?
An LLM-powered web scraper, built on Streamlit, that extracts data from pages using OpenAI, Gemini, Ollama or LiteLLM models. CyberScraper 2077 is a web scraper that uses large language models to turn web pages into structured data. You enter a URL and describe in natural language what you want, and the tool fetches the page with Playwright and asks the chosen model to pull out the matching content. Supported backends include OpenAI, Gemini, local models via Ollama, and any model exposed through a LiteLLM proxy.
Is CyberScraper-2077 free to use? What license does it have?
CyberScraper-2077 is open source under the MIT license. Check the license terms before using it in commercial products.
Is CyberScraper-2077 still maintained?
Actively maintained. The last commit to CyberScraper-2077 was on Oct 12, 2026.
What can I use CyberScraper-2077 for?
Pulling product or listing data from paginated websites into a spreadsheet. Prototyping structured extraction from pages without writing per-site selectors. Running scraping with a local Ollama model to keep data on your own machine.
How do I install CyberScraper-2077?
Clone the repository: git clone https://github.com/itsOwen/CyberScraper-2077.git Install the required packages: pip install -r requirements.txt Install Playwright: playwright install Start the Streamlit app and open http://localhost:8501: streamlit run main.py
Where can I get CyberScraper-2077?
The source code and installation instructions are on GitHub: https://github.com/itsOwen/CyberScraper-2077

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects