Repolode
Submit
muchdogesec

muchdogesec/history4feed

Builds a complete full-text historical archive of an RSS or Atom feed by pulling past posts from the Wayback Machine.

View on GitHubWebsite
Stars
182
Forks
5
Open issues
2
Last commit
2 months ago

What is it?

Most RSS and Atom feeds only list the latest handful of posts, and feed formats offer no way to page back through older entries. history4feed works around this by taking a feed URL, downloading its Wayback Machine snapshots, and collecting every unique post found in those historic copies.

For each post it downloads an HTML version of the article content and stores the post record in a database. The archived posts can then be read through an API as JSON or exposed as a new RSS feed. The project runs as a Django web server started with Docker Compose, and its Swagger UI is available for exploring the endpoints.

The project is the backend used by the web version of Obstracts, which adds features beyond this codebase.

Key features

  • 01Downloads Wayback Machine archives for a given RSS or Atom feed URL.
  • 02Identifies all unique posts across historic feed snapshots.
  • 03Fetches the full HTML content of each article and stores it in a database.
  • 04Outputs archived posts as JSON or as an RSS feed.
  • 05Provides a Django API with Swagger UI, run via Docker Compose.

Use cases

  • Recovering the full back catalogue of a long-running blog that only exposes its latest posts in its feed.
  • Building a complete, full-text archive of threat intelligence blogs for research.
  • Republishing a blog's historic posts as a single RSS feed for a reader or aggregator.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/muchdogesec/history4feed
  2. 2

    Create the .env configuration file from the template

    cp .env.example .env
  3. 3

    Build the Docker image

    sudo docker compose build
  4. 4

    Start the server and open http://127.0.0.1:8002/

    sudo docker compose up

Things to know

  • Requires Docker and a configured .env file before it runs.
  • Relies on Wayback Machine archives, so coverage depends on what was archived.

Topics

#rss#atom#wayback-machine#web-archive#django#docker#full-text

FAQ

What is history4feed?
Builds a complete full-text historical archive of an RSS or Atom feed by pulling past posts from the Wayback Machine. Most RSS and Atom feeds only list the latest handful of posts, and feed formats offer no way to page back through older entries. history4feed works around this by taking a feed URL, downloading its Wayback Machine snapshots, and collecting every unique post found in those historic copies.
Is history4feed free to use? What license does it have?
history4feed is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is history4feed still maintained?
Updated within 6 months. The last commit to history4feed was on Sep 1, 2026.
What can I use history4feed for?
Recovering the full back catalogue of a long-running blog that only exposes its latest posts in its feed. Building a complete, full-text archive of threat intelligence blogs for research. Republishing a blog's historic posts as a single RSS feed for a reader or aggregator.
How do I install history4feed?
Clone the repository: git clone https://github.com/muchdogesec/history4feed Create the .env configuration file from the template: cp .env.example .env Build the Docker image: sudo docker compose build Start the server and open http://127.0.0.1:8002/: sudo docker compose up
Where can I get history4feed?
The source code and installation instructions are on GitHub: https://github.com/muchdogesec/history4feed

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects