Repolode
Submit
beamivalice

beamivalice/sushi

A local AI inference server for Apple Silicon Macs that runs low-bit, high-quality quantized LLMs in Sushi STQ and EXL3 formats.

View on GitHub
Stars
431
Forks
36
Open issues
7
Last commit
yesterday

What is it?

Sushi is a local AI server for running low-bit quantized large language models on Apple Silicon Macs. It uses a custom Sushi Trellis Quant (STQ) format and an EXL3-Affine hybrid converter to produce compressed model packs that keep high output quality. The project is a detached fork of ddalcu's mlx-serve and can run standalone or as a guest engine inside mlx-serve. It requires an M1 or newer chip and macOS 26.2 or later.

The server listens on 127.0.0.1:12345 and supports speculative decoding (MTP for Qwen and MiMo, DFlash2 for GLM) with an 8-bit KV cache on by default. Packs cover Qwen3.8 Flash-Next, MiMo-V2.6 Flash, GLM-5.3 Flash and DeepSeek-V4.1 Flash models, with memory requirements listed from 48 GB up to 128 GB. A streaming mode keeps only part of the weights in memory and reads routed experts from the SSD, so models can run on Macs with less RAM than their weights.

The server can be exposed on a local network with an API key. The project also reports KLD and top-1 agreement figures for several packs, along with speed and agentic tool-eval results on M5 Max hardware.

Key features

  • 01Runs low-bit quantized LLMs in custom STQ and EXL3-Affine formats on Apple Silicon.
  • 02Supports speculative decoding with MTP for Qwen and MiMo, and DFlash2 for GLM.
  • 03Streams routed experts from SSD so models can run on Macs with less memory than their weights.
  • 04Defaults to an 8-bit KV cache and listens on a local port.
  • 05Can serve other computers on the network with a Bearer API key.

Use cases

  • Running large quantized coding or chat models locally on a 48 GB to 128 GB Mac.
  • Serving a local OpenAI-style endpoint to other machines on a home or office network.
  • Comparing quantization quality and speed across Qwen, GLM, MiMo and DeepSeek packs.

Quick start

  1. 1

    Install Sushi with Homebrew

    brew install beamivalice/tap/sushi
  2. 2

    Start the server with a model pack (replace <pack> with a pack name from the README)

    sushi serve <pack>

Things to know

  • Requires Apple Silicon M1 or later and macOS 26.2 or later.
  • Most model packs need 48 GB to 128 GB of memory, and some need an SSD streaming setup to run on smaller Macs.
  • The README says sushi is MIT-licensed, but the Qwen and MiMo model packs use separate licenses, and the repository's detected license is unclear.

In collections

Run LLMs locallyOpen-source Mac apps

Topics

#llm-inference#quantization#apple-silicon#local-llm#macos#speculative-decoding#zig

FAQ

What is sushi?
A local AI inference server for Apple Silicon Macs that runs low-bit, high-quality quantized LLMs in Sushi STQ and EXL3 formats. Sushi is a local AI server for running low-bit quantized large language models on Apple Silicon Macs. It uses a custom Sushi Trellis Quant (STQ) format and an EXL3-Affine hybrid converter to produce compressed model packs that keep high output quality. The project is a detached fork of ddalcu's mlx-serve and can run standalone or as a guest engine inside mlx-serve. It requires an M1 or newer chip and macOS 26.2 or later.
Is sushi free to use? What license does it have?
sushi is open source under the Other license. Check the license terms before using it in commercial products.
Is sushi still maintained?
Actively maintained. The last commit to sushi was on Nov 6, 2026.
What can I use sushi for?
Running large quantized coding or chat models locally on a 48 GB to 128 GB Mac. Serving a local OpenAI-style endpoint to other machines on a home or office network. Comparing quantization quality and speed across Qwen, GLM, MiMo and DeepSeek packs.
How do I install sushi?
Install Sushi with Homebrew: brew install beamivalice/tap/sushi Start the server with a model pack (replace <pack> with a pack name from the README): sushi serve <pack>
Where can I get sushi?
The source code and installation instructions are on GitHub: https://github.com/beamivalice/sushi

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects