Repolode
Submit
Michael-A-Kuykendall

Michael-A-Kuykendall/shimmy

A single-binary, pure-Rust inference server that runs GGUF models on any GPU through WebGPU and exposes an OpenAI-compatible API.

View on GitHub
Stars
5,988
Forks
587
Open issues
10
Last commit
4 days ago

What is it?

Shimmy is a local inference server for GGUF models that exposes OpenAI-compatible endpoints, so existing tools and SDKs can point at it without code changes. It ships as a single binary and does not need a Python runtime or a C++ toolchain.

Under the hood, Shimmy runs on Airframe, a Rust transformer engine that executes WGSL compute shaders through WebGPU. This targets NVIDIA, AMD and Intel GPUs, integrated graphics and Apple Silicon. The project lists 26 model and quantization combinations across 12 families, including Llama, Qwen, Phi, Gemma and DeepSeek-R1 distills, each passing its own certification checks.

Other notable capabilities include an INT4 KV cache option (TurboShimmy) to reduce memory use, YaRN RoPE scaling for longer context via the SHIMMY_MAX_CTX variable, and deterministic output for the same model, seed and parameters.

Key features

  • 01Serves chat completions, text completions, streaming and model listing over OpenAI-style endpoints.
  • 02Loads GGUF models directly, with model specs derived from GGUF metadata.
  • 03Runs on WebGPU across NVIDIA, AMD, Intel and Apple GPUs without backend flags.
  • 04Offers an INT4 KV cache option to reduce memory use on smaller GPUs.
  • 05Supports YaRN RoPE scaling for extended context windows.
  • 06Provides a CLI for serving and listing models.

Use cases

  • Running a private, offline chat backend for tools that already speak the OpenAI API.
  • Serving small quantized models on a consumer GPU laptop or desktop.
  • Deploying a self-contained inference binary without a Python or C++ build environment.
  • Testing applications against a local OpenAI-compatible endpoint.

Quick start

  1. 1

    Install Shimmy with Cargo

    cargo install shimmy --locked
  2. 2

    Start the server with a local GGUF model file

    shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435

Things to know

  • Only GGUF models are fully supported for native inference; SafeTensors inference is still roadmap work.
  • The README lists 26 certified model and quantization combinations, so other models may not be certified.
  • The README says MIT licensed while the metadata says Apache-2.0, so check the LICENSE file before relying on it.

Topics

#rust#webgpu#gguf#llm-inference#local-ai#openai-compatible#inference-server#api-server#command-line-tool#developer-tools

FAQ

What is shimmy?
A single-binary, pure-Rust inference server that runs GGUF models on any GPU through WebGPU and exposes an OpenAI-compatible API. Shimmy is a local inference server for GGUF models that exposes OpenAI-compatible endpoints, so existing tools and SDKs can point at it without code changes. It ships as a single binary and does not need a Python runtime or a C++ toolchain.
Is shimmy free to use? What license does it have?
shimmy is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is shimmy still maintained?
Actively maintained. The last commit to shimmy was on Oct 19, 2026.
What can I use shimmy for?
Running a private, offline chat backend for tools that already speak the OpenAI API. Serving small quantized models on a consumer GPU laptop or desktop. Deploying a self-contained inference binary without a Python or C++ build environment. Testing applications against a local OpenAI-compatible endpoint.
How do I install shimmy?
Install Shimmy with Cargo: cargo install shimmy --locked Start the server with a local GGUF model file: shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435
Where can I get shimmy?
The source code and installation instructions are on GitHub: https://github.com/Michael-A-Kuykendall/shimmy

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects