Rust libraries and Python bindings that convert LLM API formats into DeepSeek conversations and render prompts.
1.3K116RustMITUpdated yesterday
#tokenizer
Open-source projects tagged “tokenizer”
Curated GitHub repositories related to tokenizer, with AI summaries and key stats.
Sort:Recently added
2 repositories
A fast BPE tokeniser for OpenAI's models, used to convert text into tokens and back losslessly.
19.5K1.7KPythonMITUpdated 2 months ago