Open-source projects tagged “quantization”

Runs the 125B-parameter Qwen3.8-Flash-Next model on a consumer NVIDIA or AMD GPU PC, exposing OpenAI- and Anthropic-compatible APIs on localhost.

An edge-native MoE serving engine that runs large open-weight models like DeepSeek-V4 and Qwen3.6 on consumer GPUs by splitting work between GPU and CPU memory.

A minimal C11 LLM inference engine that runs 1B-class LLaMA-architecture GGUF models on low-cost boards like Raspberry Pi with about 45MB of RAM.