Repolode
Submit
mishushakov

mishushakov/llm-scraper

A TypeScript library that uses LLMs to extract structured data from any webpage, built on Playwright and Zod or JSON Schema.

View on GitHub
Stars
6,952
Forks
451
Open issues
10
Last commit
13 days ago

What is it?

LLM Scraper is a TypeScript library for pulling structured data out of web pages with large language models. You launch a Playwright browser, open a page, define the output shape with a Zod or JSON Schema, and the library sends the page content to the model and returns typed results.

The library supports several input formats: pre-processed HTML, raw HTML, markdown, extracted text via Readability.js, a screenshot for multi-modal models, or custom content produced by your own function. It works with models from OpenAI, Anthropic, Google, Groq and Ollama through the Vercel AI SDK, and version 2.0 targets AI SDK 6.

Beyond one-off extraction, it can stream partial objects as they are generated, and it can generate reusable Playwright code that performs the same scrape without calling the model again.

Key features

  • 01Extracts data using schemas defined with Zod or JSON Schema, with full TypeScript type safety.
  • 02Works with GPT, Claude, Gemini, Llama and Qwen models through the Vercel AI SDK.
  • 03Supports six input modes: html, raw_html, markdown, text, image and custom.
  • 04Streams partial results as the model generates them.
  • 05Generates reusable Playwright code for a given schema.

Use cases

  • Turning product listings or news front pages into JSON for a database or dashboard.
  • Scraping sites that lack an API and whose layout changes often, where selectors would break.
  • Prototyping data extraction with different LLM providers and comparing results.
  • Generating a fixed Playwright script from a schema to avoid repeated model calls.

Quick start

  1. 1

    Install the core dependencies from npm

    npm i zod playwright llm-scraper
  2. 2

    Install the AI SDK provider for your LLM (e.g. OpenAI)

    npm i @ai-sdk/openai
  3. 3

    Initialize the LLM and create a scraper instance, then run it on a page

Things to know

  • Extraction quality and cost depend on the LLM provider you choose, which usually requires an API key.

In collections

Browser automation for AI agents

Topics

#web-scraping#llm#playwright#typescript#structured-data#browser-automation#ai#artificial-intelligence#browser#gpt

FAQ

What is llm-scraper?
A TypeScript library that uses LLMs to extract structured data from any webpage, built on Playwright and Zod or JSON Schema. LLM Scraper is a TypeScript library for pulling structured data out of web pages with large language models. You launch a Playwright browser, open a page, define the output shape with a Zod or JSON Schema, and the library sends the page content to the model and returns typed results.
Is llm-scraper free to use? What license does it have?
llm-scraper is open source under the MIT license. Check the license terms before using it in commercial products.
Is llm-scraper still maintained?
Actively maintained. The last commit to llm-scraper was on Oct 26, 2026.
What can I use llm-scraper for?
Turning product listings or news front pages into JSON for a database or dashboard. Scraping sites that lack an API and whose layout changes often, where selectors would break. Prototyping data extraction with different LLM providers and comparing results. Generating a fixed Playwright script from a schema to avoid repeated model calls.
How do I install llm-scraper?
Install the core dependencies from npm: npm i zod playwright llm-scraper Install the AI SDK provider for your LLM (e.g. OpenAI): npm i @ai-sdk/openai Initialize the LLM and create a scraper instance, then run it on a page
Where can I get llm-scraper?
The source code and installation instructions are on GitHub: https://github.com/mishushakov/llm-scraper

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects