Repolode
Submit
codelucas

codelucas/newspaper

Python 3 library that downloads news articles and extracts their text, authors, publish date, images, keywords and summaries.

View on GitHubWebsite
Stars
15,180
Forks
2,107
Open issues
515
Last commit
2 days ago

What is it?

newspaper3k is a Python 3 library for pulling news articles from URLs and turning raw HTML into structured data. Given an article URL, it downloads the page, then extracts the title, body text, authors, publish date, top image, embedded videos, and NLP-derived keywords and summary.

Beyond single articles, it can crawl a news site's homepage and category pages to discover article URLs and list the site's categories. A separate fulltext function works on HTML you have already fetched. Language is auto-detected unless you specify one, and the library supports a list of more than 35 languages, including Chinese, Japanese, Arabic and Russian.

The project is installed with pip under the name newspaper3k and is released under the MIT license. It is the Python 3 successor to the original newspaper package.

Key features

  • 01Extracts article title, body text, authors and publish date from a URL
  • 02Finds top and all images, plus embedded videos, in an article
  • 03Generates keywords and summaries using built-in NLP
  • 04Discovers article links and categories by crawling a news site
  • 05Auto-detects language and supports more than 35 languages

Use cases

  • Building a news aggregator or monitoring pipeline that collects articles from many sites
  • Extracting clean article text for NLP, search indexing or dataset creation
  • Archiving articles with their metadata and images
  • Processing Chinese or other non-English news content

Quick start

  1. 1

    Install the package with pip

    pip3 install newspaper3k
  2. 2

    Run the official corpora download script from the README (NLP features need it)

Things to know

  • On Debian/Ubuntu and other Linux systems, extra system libraries such as libxml2 and libxslt must be installed first.
  • NLP features such as keywords and summaries need an extra corpora download step.

Topics

#python#web-scraping#news-extraction#article-parsing#nlp#crawler#crawling#news#news-aggregator#scraper

FAQ

What is newspaper?
Python 3 library that downloads news articles and extracts their text, authors, publish date, images, keywords and summaries. newspaper3k is a Python 3 library for pulling news articles from URLs and turning raw HTML into structured data. Given an article URL, it downloads the page, then extracts the title, body text, authors, publish date, top image, embedded videos, and NLP-derived keywords and summary.
Is newspaper free to use? What license does it have?
newspaper is open source under the MIT license. Check the license terms before using it in commercial products.
Is newspaper still maintained?
Actively maintained. The last commit to newspaper was on Oct 19, 2026.
What can I use newspaper for?
Building a news aggregator or monitoring pipeline that collects articles from many sites Extracting clean article text for NLP, search indexing or dataset creation Archiving articles with their metadata and images Processing Chinese or other non-English news content
How do I install newspaper?
Install the package with pip: pip3 install newspaper3k Run the official corpora download script from the README (NLP features need it)
Where can I get newspaper?
The source code and installation instructions are on GitHub: https://github.com/codelucas/newspaper

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects