Open-source projects tagged “speculative-decoding”

A launcher and patched llama.cpp setup that runs the 1.58-bit Ternary Bonsai 2 27B model with a 262k context on 12 GB and 8 GB NVIDIA GPUs.

Pairs an iPhone with a Mac over USB-C so the phone's GPU helps run a Qwen 27B model locally, speeding up prompt reading and extending context.

Open source inference engine that tunes its kernels to your hardware to run open models faster on Apple Silicon, NVIDIA, AMD, or CPU.