Repolode
Submit
huggingface

huggingface/tokenizers

Fast, state-of-the-art tokenization library written in Rust, with Python and Node.js bindings, for NLP and language model pipelines.

View on GitHubWebsite
Stars
11,183
Forks
1,226
Open issues
198
Last commit
6 hours ago

What is it?

tokenizers is a tokenization library from Hugging Face for turning text into token IDs and back. It implements the common subword models (BPE, Unigram, WordPiece, WordLevel) along with the normalizer, pre-tokenizer, post-processor and decoder stages that make up a full tokenization pipeline. Developers can load a pretrained tokenizer with from_pretrained or from a tokenizer.json file, then encode and decode text from Python or Node.js.

The project is split into sub-crates so that a serving binary can include only the inference path and skip training, legacy format conversion, or the bitstream pre-tokenizer. The current release candidate is v1.0.0.rc.0, installed with pip install --pre tokenizers. Some Python APIs from earlier versions, such as training and offset output, are listed as being brought back before the final v1.0 release.

The Rust core has a portable scalar path and hardware-specific kernels for NEON, SSE, AVX and WebAssembly SIMD, with runtime dispatch on x86_64.

Key features

  • 01Implements BPE, Unigram, WordPiece and WordLevel tokenization models.
  • 02Provides Python and Node.js bindings on top of a Rust core.
  • 03Loads pretrained tokenizers from the Hugging Face Hub or a local tokenizer.json file.
  • 04Exposes encode, decode and tokenize methods with attention masks and type IDs.
  • 05Splits into sub-crates so serving builds can exclude training code.

Use cases

  • Preparing text inputs for transformer models such as BERT or GPT-style LLMs.
  • Running tokenization inside a production inference service with a small dependency footprint.
  • Training a custom tokenizer on a domain corpus.
  • Converting or reading legacy tokenizer.json files in a Node.js or Python application.

Topics

#nlp#tokenization#rust#python#language-model#transformers#bpe#wordpiece#bert#gpt

FAQ

What is tokenizers?
Fast, state-of-the-art tokenization library written in Rust, with Python and Node.js bindings, for NLP and language model pipelines. tokenizers is a tokenization library from Hugging Face for turning text into token IDs and back. It implements the common subword models (BPE, Unigram, WordPiece, WordLevel) along with the normalizer, pre-tokenizer, post-processor and decoder stages that make up a full tokenization pipeline. Developers can load a pretrained tokenizer with from_pretrained or from a tokenizer.json file, then encode and decode text from Python or Node.js.
Is tokenizers free to use? What license does it have?
tokenizers is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is tokenizers still maintained?
Actively maintained. The last commit to tokenizers was on Oct 18, 2026.
What can I use tokenizers for?
Preparing text inputs for transformer models such as BERT or GPT-style LLMs. Running tokenization inside a production inference service with a small dependency footprint. Training a custom tokenizer on a domain corpus. Converting or reading legacy tokenizer.json files in a Node.js or Python application.
Where can I get tokenizers?
The source code and installation instructions are on GitHub: https://github.com/huggingface/tokenizers

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects