Repolode
Submit
JustVugg

JustVugg/colibri

A pure C inference engine that runs very large mixture-of-experts models on ordinary hardware by streaming routed experts from disk.

View on GitHubWebsite
Stars
42,807
Forks
4,789
Open issues
151
Last commit
9 hours ago

What is it?

colibri runs very large open mixture-of-experts language models on the machine you already have. A MoE model with hundreds of billions of parameters activates only a small part of itself per token, so colibri keeps the dense part in RAM and reads the routed experts from disk as the router requests them, through a cache that learns which experts your workload uses. A GPU is optional; when present, it can hold the hottest experts and dense layers.

The engine is written in pure C with one source file per model family and no runtime dependencies, and it supports a range of models including GLM-5.2, DeepSeek V4 Flash, Qwen3.6, Kimi K3, and OLMoE, plus an image model (Qwen-Image-2.1) and decision models. It exposes OpenAI- and Anthropic-compatible HTTP APIs, a terminal chat, and a web dashboard. A one-step setup script detects hardware, recommends a model that fits in RAM and disk, builds the engine, and downloads the weights.

Performance depends heavily on hardware: decode speed ranges from a fraction of a token per second on a small laptop with models streaming from disk to several tokens per second with resident experts on GPUs, and the project publishes measured numbers per machine.

Key features

  • 01Runs MoE models larger than RAM by streaming routed experts from disk through an adaptive cache.
  • 02Written in pure C with no runtime dependencies and no GPU required.
  • 03Supports CPU, Vulkan, CUDA on Linux and Windows, and Metal on macOS.
  • 04Provides OpenAI- and Anthropic-compatible APIs, a terminal chat, and a web dashboard.
  • 05One-step setup detects hardware and recommends a model that fits.

Use cases

  • Running a frontier-scale open MoE model on a workstation or laptop with limited GPU memory.
  • Studying MoE inference internals, expert routing, and memory placement.
  • Serving a local OpenAI-compatible endpoint for coding tools backed by a large open model.
  • Running structured yes/no or multi-choice decisions with calibrated probabilities.

Quick start

  1. 1

    Install git, Python and build tools on Ubuntu/Debian

    sudo apt install git python3 build-essential
  2. 2

    Clone the repository

    git clone https://github.com/JustVugg/colibri
  3. 3

    Enter the folder and run the setup, which picks a model and starts colibri

    cd colibri && ./start-here.sh

Things to know

  • Very large models need 16 GB or more RAM and tens to hundreds of GB of disk, and the largest need over 1 TB.
  • Large models stream experts from disk, so speed is slow (often under 1 token per second) and depends on disk speed.
  • Qwen-Image-2.1 is licensed for non-commercial use only.

Topics

#llm-inference#mixture-of-experts#local-llm#vulkan#cuda#openai-compatible#offloading

FAQ

What is colibri?
A pure C inference engine that runs very large mixture-of-experts models on ordinary hardware by streaming routed experts from disk. colibri runs very large open mixture-of-experts language models on the machine you already have. A MoE model with hundreds of billions of parameters activates only a small part of itself per token, so colibri keeps the dense part in RAM and reads the routed experts from disk as the router requests them, through a cache that learns which experts your workload uses. A GPU is optional; when present, it can hold the hottest experts and dense layers.
Is colibri free to use? What license does it have?
colibri is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is colibri still maintained?
Actively maintained. The last commit to colibri was on Oct 19, 2026.
What can I use colibri for?
Running a frontier-scale open MoE model on a workstation or laptop with limited GPU memory. Studying MoE inference internals, expert routing, and memory placement. Serving a local OpenAI-compatible endpoint for coding tools backed by a large open model. Running structured yes/no or multi-choice decisions with calibrated probabilities.
How do I install colibri?
Install git, Python and build tools on Ubuntu/Debian: sudo apt install git python3 build-essential Clone the repository: git clone https://github.com/JustVugg/colibri Enter the folder and run the setup, which picks a model and starts colibri: cd colibri && ./start-here.sh
Where can I get colibri?
The source code and installation instructions are on GitHub: https://github.com/JustVugg/colibri

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects