Repolode
Submit
drumih

drumih/turbo-fieldfare

Swift and Metal runtime that runs the 26B Gemma 4 26B-A4B model on Apple Silicon Macs with about 2 GB of RAM by streaming experts from SSD.

View on GitHub
Stars
6,944
Forks
461
Open issues
53
Last commit
5 days ago

What is it?

TurboFieldfare runs the instruction-tuned Gemma 4 26B-A4B model without loading all 14.3 GB of weights into memory. It keeps the shared core weights and FP16 KV cache resident, then streams only the routed experts needed for each token from SSD through a bounded expert cache. This lets the model run on Apple Silicon Macs with 8 GB of RAM.

The project is a model-specific runtime written in Swift and Metal rather than a wrapper around MLX or llama.cpp. It includes a streaming installer that repacks the pinned Hugging Face checkpoint into a local .gturbo format, a native Mac app, a command-line interface, and a loopback OpenAI-compatible Chat Completions server. Image input is available through an optional companion pack on M2 or newer chips.

The project requires macOS 26, Metal 4, and Swift 6.2, and only supports arm64 Macs. Reported decode speeds are 5.1-6.3 tokens per second on an 8 GB M2 MacBook Air and 31-35 tokens per second on a 24 GB M5 Pro.

Key features

  • 01Streams routed experts from SSD through a bounded cache, keeping about 2 GB of weights and KV cache in memory.
  • 02Runs on Apple Silicon Macs with 8 GB of RAM, validated on an 8 GB M2 MacBook Air.
  • 03Includes a native Mac app, a CLI, and a loopback OpenAI-compatible Chat Completions server.
  • 04Streaming installer repacks the Hugging Face checkpoint directly without staging a full copy on disk.
  • 05Supports function-tool declarations in the server, with tool calls returned for the client to execute.
  • 06Optional companion pack adds image input on M2 or newer chips.

Use cases

  • Running a large Gemma model locally on a base 8 GB MacBook without cloud access.
  • Experimenting with expert streaming and Metal kernel design on Apple Silicon.
  • Serving a local OpenAI-compatible endpoint for tools that expect the Chat Completions API.
  • Chatting with a private, on-device model from a native Mac app.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/drumih/turbo-fieldfare.git
  2. 2

    Enter the project directory

    cd turbo-fieldfare
  3. 3

    Build the release version

    swift build -c release
  4. 4

    Launch the Mac app, then download and load the model inside it

    .build/release/TurboFieldfareMac

Things to know

  • Requires an Apple Silicon Mac running macOS 26 with Metal 4; Intel and older macOS are not supported.
  • The first run downloads and installs about 15 GB of model weights.
  • Throughput varies widely by hardware and is slow on an 8 GB M2 (about 5-6 tok/s).
  • The local server has no remote authentication or TLS and must stay on loopback.

Topics

#apple-silicon#metal#llm-inference#on-device-ai#gemma#swift#local-ai#gemma4#gemma4-26b-a4b#gpgpu

FAQ

What is turbo-fieldfare?
Swift and Metal runtime that runs the 26B Gemma 4 26B-A4B model on Apple Silicon Macs with about 2 GB of RAM by streaming experts from SSD. TurboFieldfare runs the instruction-tuned Gemma 4 26B-A4B model without loading all 14.3 GB of weights into memory. It keeps the shared core weights and FP16 KV cache resident, then streams only the routed experts needed for each token from SSD through a bounded expert cache. This lets the model run on Apple Silicon Macs with 8 GB of RAM.
Is turbo-fieldfare free to use? What license does it have?
turbo-fieldfare is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is turbo-fieldfare still maintained?
Actively maintained. The last commit to turbo-fieldfare was on Oct 15, 2026.
What can I use turbo-fieldfare for?
Running a large Gemma model locally on a base 8 GB MacBook without cloud access. Experimenting with expert streaming and Metal kernel design on Apple Silicon. Serving a local OpenAI-compatible endpoint for tools that expect the Chat Completions API. Chatting with a private, on-device model from a native Mac app.
How do I install turbo-fieldfare?
Clone the repository: git clone https://github.com/drumih/turbo-fieldfare.git Enter the project directory: cd turbo-fieldfare Build the release version: swift build -c release Launch the Mac app, then download and load the model inside it: .build/release/TurboFieldfareMac
Where can I get turbo-fieldfare?
The source code and installation instructions are on GitHub: https://github.com/drumih/turbo-fieldfare

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects