Repolode
Enviar
JustVugg

JustVugg/colibri

A pure C inference engine that runs very large mixture-of-experts models on ordinary hardware by streaming routed experts from disk.

Ver en GitHubSitio web
Estrellas
42,807
Forks
4,789
Issues abiertos
151
Último commit
hace 2 horas

¿Qué es?

colibri runs very large open mixture-of-experts language models on the machine you already have. A MoE model with hundreds of billions of parameters activates only a small part of itself per token, so colibri keeps the dense part in RAM and reads the routed experts from disk as the router requests them, through a cache that learns which experts your workload uses. A GPU is optional; when present, it can hold the hottest experts and dense layers.

The engine is written in pure C with one source file per model family and no runtime dependencies, and it supports a range of models including GLM-5.2, DeepSeek V4 Flash, Qwen3.6, Kimi K3, and OLMoE, plus an image model (Qwen-Image-2.1) and decision models. It exposes OpenAI- and Anthropic-compatible HTTP APIs, a terminal chat, and a web dashboard. A one-step setup script detects hardware, recommends a model that fits in RAM and disk, builds the engine, and downloads the weights.

Performance depends heavily on hardware: decode speed ranges from a fraction of a token per second on a small laptop with models streaming from disk to several tokens per second with resident experts on GPUs, and the project publishes measured numbers per machine.

Funciones clave

  • 01Runs MoE models larger than RAM by streaming routed experts from disk through an adaptive cache.
  • 02Written in pure C with no runtime dependencies and no GPU required.
  • 03Supports CPU, Vulkan, CUDA on Linux and Windows, and Metal on macOS.
  • 04Provides OpenAI- and Anthropic-compatible APIs, a terminal chat, and a web dashboard.
  • 05One-step setup detects hardware and recommends a model that fits.

Casos de uso

  • Running a frontier-scale open MoE model on a workstation or laptop with limited GPU memory.
  • Studying MoE inference internals, expert routing, and memory placement.
  • Serving a local OpenAI-compatible endpoint for coding tools backed by a large open model.
  • Running structured yes/no or multi-choice decisions with calibrated probabilities.

Temas

#llm-inference#mixture-of-experts#local-llm#vulkan#cuda#openai-compatible#offloading

Preguntas frecuentes

¿Qué es colibri?
A pure C inference engine that runs very large mixture-of-experts models on ordinary hardware by streaming routed experts from disk. colibri runs very large open mixture-of-experts language models on the machine you already have. A MoE model with hundreds of billions of parameters activates only a small part of itself per token, so colibri keeps the dense part in RAM and reads the routed experts from disk as the router requests them, through a cache that learns which experts your workload uses. A GPU is optional; when present, it can hold the hottest experts and dense layers.
¿colibri es gratuito? ¿Qué licencia tiene?
colibri es open source bajo la licencia Apache-2.0. Revisa los términos de la licencia antes de usarlo en productos comerciales.
¿colibri sigue mantenido?
Mantenimiento activo. El último commit de colibri fue el 19 oct 2026.
¿Para qué puedo usar colibri?
Running a frontier-scale open MoE model on a workstation or laptop with limited GPU memory. Studying MoE inference internals, expert routing, and memory placement. Serving a local OpenAI-compatible endpoint for coding tools backed by a large open model. Running structured yes/no or multi-choice decisions with calibrated probabilities.
¿Dónde consigo colibri?
El código fuente y las instrucciones de instalación están en GitHub: https://github.com/JustVugg/colibri

Resumen generado por IA a partir del README del proyecto. Las estadísticas de GitHub se actualizan a diario.

Proyectos relacionados