Repolode
Submit
spencermountain

spencermountain/wtf_wikipedia

A JavaScript library that parses Wikipedia wikitext into plaintext, structured JSON, infoboxes, links, and images.

View on GitHubWebsite
Stars
858
Forks
130
Open issues
54
Last commit
3 days ago

What is it?

wtf_wikipedia turns raw Wikipedia wikitext into usable data. It handles the templates, nested links, references and citations that make wiki markup hard to parse, and returns a document object with methods for plaintext, sections, sentences, links, categories, infoboxes and images.

The library can fetch pages directly from the MediaWiki API, including non-English Wikipedias and third-party wikis such as Fandom. Output can be exported as plaintext or as JSON in small, medium or large presets. Redirects and disambiguation pages are detected, and dynamic templates and geo-coordinates are converted to readable values.

It works in Node.js and in the browser, ships TypeScript declarations, and can be extended with custom template parsers and plugins. The project notes that it is not the fastest parser and that transcluded page data and perfect formatting of infobox HTML are not supported.

Key features

  • 01Parses wikitext into plaintext, sections, sentences, and paragraphs.
  • 02Extracts infobox key-value data, links, images, and categories.
  • 03Fetches pages from Wikipedia in any language and from third-party MediaWiki wikis.
  • 04Exports page data as JSON in small, medium, or large presets.
  • 05Supports custom template parsers and plugins, with TypeScript types included.

Use cases

  • Building a dataset of structured facts, such as sports rosters or astronaut lists, from Wikipedia tables and infoboxes.
  • Extracting clean article text for NLP or search indexing.
  • Processing a full Wikipedia dump offline with the companion dumpster-dive tool.
  • Pulling article data from a Fandom or other MediaWiki-based wiki.

Quick start

  1. 1

    Install the package from npm

    npm install wtf_wikipedia
  2. 2

    Import it and fetch a Wikipedia article

  3. 3

    Read plaintext from the parsed document

Things to know

  • Does not fetch transcluded page data from external pages.
  • Does not preserve perfect page order or maintain template CSS styling.
  • Parsing wikitext is inherently imperfect, so some pages may not render correctly.

In collections

Natural language processing tools & libraries

Topics

#wikipedia#wikitext#parser#mediawiki#nlp#text-extraction#typescript#wikipedia-markup-parser

FAQ

What is wtf_wikipedia?
A JavaScript library that parses Wikipedia wikitext into plaintext, structured JSON, infoboxes, links, and images. wtf_wikipedia turns raw Wikipedia wikitext into usable data. It handles the templates, nested links, references and citations that make wiki markup hard to parse, and returns a document object with methods for plaintext, sections, sentences, links, categories, infoboxes and images.
Is wtf_wikipedia free to use? What license does it have?
wtf_wikipedia is open source under the MIT license. Check the license terms before using it in commercial products.
Is wtf_wikipedia still maintained?
Actively maintained. The last commit to wtf_wikipedia was on Nov 4, 2026.
What can I use wtf_wikipedia for?
Building a dataset of structured facts, such as sports rosters or astronaut lists, from Wikipedia tables and infoboxes. Extracting clean article text for NLP or search indexing. Processing a full Wikipedia dump offline with the companion dumpster-dive tool. Pulling article data from a Fandom or other MediaWiki-based wiki.
How do I install wtf_wikipedia?
Install the package from npm: npm install wtf_wikipedia Import it and fetch a Wikipedia article Read plaintext from the parsed document
Where can I get wtf_wikipedia?
The source code and installation instructions are on GitHub: https://github.com/spencermountain/wtf_wikipedia

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects