Repolode
Submit
openai

openai/tiktoken

A fast BPE tokeniser for OpenAI's models, used to convert text into tokens and back losslessly.

View on GitHub
Stars
19,493
Forks
1,653
Open issues
144
Last commit
2 months ago

What is it?

tiktoken is a byte pair encoding (BPE) tokeniser built for OpenAI's models. It converts text into the sequence of integer tokens a language model reads, and decodes tokens back into the original text. Developers typically use it to count tokens before sending a prompt, so they can estimate cost or stay within a model's context limit.

You load an encoding by name, such as o200k_base, or ask for the encoding used by a specific model like gpt-4o. The Python package is installed from PyPI. The project also includes an educational submodule that shows how BPE training and encoding work step by step.

The package can be extended with custom encodings, either by constructing an Encoding object directly or by registering encodings through a tiktoken_ext plugin namespace package.

Key features

  • 01Encodes text into tokens and decodes tokens back to the original text losslessly.
  • 02Provides named encodings such as o200k_base and cl100k_base.
  • 03Looks up the encoding used by a specific OpenAI model via encoding_for_model.
  • 04Can be installed from PyPI with pip.
  • 05Allows custom encodings through direct construction or a plugin mechanism.

Use cases

  • Counting tokens in a prompt to estimate API usage before sending a request.
  • Checking whether input fits within a model's context window.
  • Learning how BPE tokenisation works using the bundled educational module.

Quick start

  1. 1

    Install the package from PyPI

    pip install tiktoken
  2. 2

    Load an encoding and round-trip text in Python

    python -c "import tiktoken; enc = tiktoken.get_encoding('o200k_base'); assert enc.decode(enc.encode('hello world')) == 'hello world'"

Things to know

  • The README documents only the Python package; it does not describe other language bindings.

Topics

#tokenizer#bpe#nlp#llm#openai#python

FAQ

What is tiktoken?
A fast BPE tokeniser for OpenAI's models, used to convert text into tokens and back losslessly. tiktoken is a byte pair encoding (BPE) tokeniser built for OpenAI's models. It converts text into the sequence of integer tokens a language model reads, and decodes tokens back into the original text. Developers typically use it to count tokens before sending a prompt, so they can estimate cost or stay within a model's context limit.
Is tiktoken free to use? What license does it have?
tiktoken is open source under the MIT license. Check the license terms before using it in commercial products.
Is tiktoken still maintained?
Updated within 6 months. The last commit to tiktoken was on Aug 17, 2026.
What can I use tiktoken for?
Counting tokens in a prompt to estimate API usage before sending a request. Checking whether input fits within a model's context window. Learning how BPE tokenisation works using the bundled educational module.
How do I install tiktoken?
Install the package from PyPI: pip install tiktoken Load an encoding and round-trip text in Python: python -c "import tiktoken; enc = tiktoken.get_encoding('o200k_base'); assert enc.decode(enc.encode('hello world')) == 'hello world'"
Where can I get tiktoken?
The source code and installation instructions are on GitHub: https://github.com/openai/tiktoken

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects