Open-source projects tagged “llm-inference”

Open-source Go/Python runtime for serverless AI workloads, offering GPU inference endpoints, sandboxes and background task queues with scale-to-zero.

Swift and Metal runtime that runs Gemma 4 26B-A4B on Apple Silicon Macs with about 2 GB of RAM by streaming experts from SSD.

An edge-native MoE serving engine that runs large open-weight models like DeepSeek-V4 and Qwen3.6 on consumer GPUs by splitting work between GPU and CPU memory.

A pure C inference engine that runs very large mixture-of-experts models on ordinary hardware by streaming routed experts from disk.

A small educational project for running vision-language model inference on Apple Silicon with MLX, scoring candidate answers about one image.

Uses an iPhone connected over USB-C to speed up prefill and extend context for a 27B Qwen model running locally on a Mac.