C++ and Python library that runs Transformer models such as Whisper, Llama and NLLB faster and with less memory via quantization and optimized kernels.
4.7K+7 this week552C++MITWell maintainedUpdated 4 hours ago
FareedKhan-dev/kimi-k3-in-c
A C99 inference engine that runs the 2.78-trillion-parameter Kimi K3 model on a single CPU, streaming experts from disk with no GPU or BLAS.
- Stars
- 11,187
- Forks
- 1,739
- Open issues
- 4
- Last commit
- last month
What is it?
kimi-k3-in-c is an inference engine for Moonshot's Kimi K3 mixture-of-experts model, written in portable C99 with no BLAS, framework or GPU dependency. The checkpoint is 1.56 TB on disk, but only 16 of 896 routed experts fire per layer for any token, so the engine keeps the dense trunk in memory and streams the rest from disk. The project reports a peak resident set of 8.24 GB on an 8 GB-class setup, with output that is byte-identical across memory budgets; more RAM only makes generation faster.
The engine covers the full pipeline: a safetensors reader, a config reader, a tokenizer that matches the official one, MXFP4 expert kernels, KDA and MLA attention, an LRU cache for experts, and a text chat mode using the official XTML format. Setup involves downloading the checkpoint, packing the trunk into a single file, and choosing a memory preset. A test suite checks the engine against a PyTorch reference on a small model without needing the full weights.
The project is aimed at developers who want to study or run very large MoE models on ordinary hardware, and it is useful as a reference for low-level inference design. Generation speed is bounded by disk throughput, so a local NVMe drive matters a great deal.
Key features
- 01Runs the 2.78T-parameter Kimi K3 on one CPU with about 8 GB of peak RAM.
- 02Streams routed experts from disk and caches hot ones in an LRU cache.
- 03Uses MXFP4 kernels and AVX2 or NEON SIMD paths with no BLAS dependency.
- 04Produces byte-identical output at every memory budget.
- 05Includes a text chat mode using the official XTML format and tokenizer.
- 06Ships a test suite that checks outputs against a PyTorch reference.
Use cases
- Running a very large MoE model on a laptop or desktop without a GPU.
- Studying how streamed mixture-of-experts inference and KV or recurrent state work at the C level.
- Using it as a reference when building a CPU inference engine for other large models.
- Benchmarking how storage throughput limits CPU-based LLM generation.
Quick start
- 1
Clone the repository
git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git - 2
Enter the project directory
cd kimi-k3-in-c - 3
Build the engine
make -j - 4
Run the test suite (no checkpoint needed)
make test
Things to know
- Running the full model requires a 1.56 TB checkpoint on disk, about 1.7 TB free space in total.
- Generation is very slow on small machines, around 26 seconds per token with 8 GB of RAM.
- Chat mode and the full model require downloading the large official checkpoint separately.
In collections
Topics
#llm-inference#cpu-inference#mixture-of-experts#quantization#c99#simd#model-streaming#avx2#deep-learning#from-scratch
FAQ
- What is kimi-k3-in-c?
- A C99 inference engine that runs the 2.78-trillion-parameter Kimi K3 model on a single CPU, streaming experts from disk with no GPU or BLAS. kimi-k3-in-c is an inference engine for Moonshot's Kimi K3 mixture-of-experts model, written in portable C99 with no BLAS, framework or GPU dependency. The checkpoint is 1.56 TB on disk, but only 16 of 896 routed experts fire per layer for any token, so the engine keeps the dense trunk in memory and streams the rest from disk. The project reports a peak resident set of 8.24 GB on an 8 GB-class setup, with output that is byte-identical across memory budgets; more RAM only makes generation faster.
- Is kimi-k3-in-c free to use? What license does it have?
- kimi-k3-in-c is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
- Is kimi-k3-in-c still maintained?
- Updated within 6 months. The last commit to kimi-k3-in-c was on Oct 2, 2026.
- What can I use kimi-k3-in-c for?
- Running a very large MoE model on a laptop or desktop without a GPU. Studying how streamed mixture-of-experts inference and KV or recurrent state work at the C level. Using it as a reference when building a CPU inference engine for other large models. Benchmarking how storage throughput limits CPU-based LLM generation.
- How do I install kimi-k3-in-c?
- Clone the repository: git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git Enter the project directory: cd kimi-k3-in-c Build the engine: make -j Run the test suite (no checkpoint needed): make test
- Where can I get kimi-k3-in-c?
- The source code and installation instructions are on GitHub: https://github.com/FareedKhan-dev/kimi-k3-in-c
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
A lightweight, standalone C++ inference engine for running Google's Gemma models on CPU.
7.1K+8 this week672C++Apache-2.0Updated 5 hours ago
Llama 2 style transformer inference written in pure Mojo, in a single file, using SIMD vectorization and multi-threading.
2.1K139MojoMITUpdated last month
A research framework for CPU-GPU heterogeneous LLM inference and LoRA/full fine-tuning of large MoE models on limited GPU memory.
19.7K+12 this week1.6KPythonApache-2.0Well maintainedUpdated 2 days ago
A local inference engine that runs the Qwen3.8-Flash-Next mixture-of-experts model on NVIDIA Blackwell GPUs, served through OpenAI- and Anthropic-compatible APIs.
750+370 this week98C++MITUpdated 2 hours ago
A PyTorch-based toolkit of from-scratch LLM implementations for pretraining, finetuning, evaluating and serving 20+ open-weight models.
13.7K+8 this week1.5KPythonApache-2.0Well maintainedUpdated 6 days ago