Repolode
Submit
adbar

adbar/trafilatura

Python package and CLI for crawling websites and extracting main text, metadata and comments from raw HTML.

Open-source alternative to:Diffbot

View on GitHubWebsite
Stars
6,995
Forks
460
Open issues
92
Last commit
14 days ago

What is it?

Trafilatura is a Python package and command-line tool for collecting text from the web. It covers discovery (sitemaps, feeds and URL filtering), downloading, and extraction of the main text, metadata such as title, author and date, and optionally comments, links, images and tables. The goal is to keep boilerplate such as headers, footers and navigation out of the output while retaining the actual article content.

The extractor is rule-based and uses jusText and readability-lxml as fallbacks. Results can be written as TXT, Markdown, CSV, JSON, HTML, XML or XML-TEI. No database is required, so it fits into scripts and data pipelines directly.

The library is used for building text corpora and NLP datasets, and it supports language detection and an R interface in addition to Python.

Key features

  • 01Discovers URLs through sitemaps and RSS, Atom or JSON feeds, with filtering and deduplication.
  • 02Extracts main text, title, author, date, categories and tags from HTML.
  • 03Optionally extracts comments, links, images and tables.
  • 04Outputs TXT, Markdown, CSV, JSON, HTML, XML and XML-TEI.
  • 05Works on live URLs as well as previously downloaded HTML files.
  • 06Available as both a Python library and a command-line tool.

Use cases

  • Building text corpora for NLP or linguistic research.
  • Collecting news articles from websites and feeds for analysis.
  • Preparing clean web text as input for LLM or RAG pipelines.
  • Scraping article content without writing site-specific parsers.

Quick start

  1. 1

    Install Trafilatura following the installation guide in the documentation

  2. 2

    Import fetch_url and extract, download a page and extract its text and metadata using the README Python example

Things to know

  • Versions prior to v1.8.0 are licensed under GPLv3+ rather than Apache 2.0.

Topics

#web-scraping#text-extraction#crawler#nlp#corpus-builder#html-to-markdown#readability#article-extractor#corpus-tools#html2text

FAQ

What is trafilatura?
Python package and CLI for crawling websites and extracting main text, metadata and comments from raw HTML. Trafilatura is a Python package and command-line tool for collecting text from the web. It covers discovery (sitemaps, feeds and URL filtering), downloading, and extraction of the main text, metadata such as title, author and date, and optionally comments, links, images and tables. The goal is to keep boilerplate such as headers, footers and navigation out of the output while retaining the actual article content.
Is trafilatura free to use? What license does it have?
trafilatura is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is trafilatura still maintained?
Actively maintained. The last commit to trafilatura was on Oct 6, 2026.
What can I use trafilatura for?
Building text corpora for NLP or linguistic research. Collecting news articles from websites and feeds for analysis. Preparing clean web text as input for LLM or RAG pipelines. Scraping article content without writing site-specific parsers.
How do I install trafilatura?
Install Trafilatura following the installation guide in the documentation Import fetch_url and extract, download a page and extract its text and metadata using the README Python example
Where can I get trafilatura?
The source code and installation instructions are on GitHub: https://github.com/adbar/trafilatura

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects