Repolode
Submit
carloslfu

carloslfu/slotstream

A native Swift engine that runs a 105 GB Qwen3.8-Flash-Next 4-bit model on 16–64 GB Apple Silicon Macs by streaming experts from SSD.

View on GitHub
Stars
652
Forks
54
Open issues
77
Last commit
2 days ago

What is it?

Slotstream is a local inference engine for Apple Silicon Macs, written in Swift on Apple's MLX and Metal. It runs a mixture-of-experts model whose weights are far larger than system RAM: shared weights stay in memory, while the experts needed for each token are read from SSD into a bounded cache, with frequently used experts kept in RAM. Inference runs entirely on the Mac after the model is downloaded, with no Python runtime or cloud account.

The released engine targets Qwen3.8-Flash-Next 4-bit on Macs with 16 to 64 GB of memory. Auto mode chooses the memory budget, cache size, speculative decoding and context window at startup, and a governor shrinks the cache when other apps need memory. The project provides a CLI, Ollama-, OpenAI- and Anthropic-compatible HTTP APIs, and a Swift library. The `slotstream launch` command starts Claude Code, Codex, Pi, opencode or Hermes already connected to the local server.

The engine requires macOS 14 or later and Apple Silicon; Windows, Linux and Intel Macs are not supported.

Key features

  • 01Streams mixture-of-experts weights from SSD into a bounded in-memory expert cache.
  • 02Automatic memory planning picks cache size, context window and speculative decoding for the Mac.
  • 03Exposes Ollama-, OpenAI- and Anthropic-compatible APIs for local apps.
  • 04Launches Claude Code, Codex, Pi, opencode and Hermes against the local model with one command.
  • 05Provides a Swift library for embedding the engine in native apps.
  • 06Runs fully offline after the model download.

Use cases

  • Running a large open-weight model on a 16–64 GB Mac that cannot hold it in RAM.
  • Using Claude Code or Codex with a local model so code stays on your machine.
  • Connecting a local chat front end such as Open WebUI to an offline model server.
  • Embedding a local LLM in a native macOS app written in Swift.

Quick start

  1. 1

    Run the official install script from the README

  2. 2

    Check your Mac's memory and disk space

    slotstream doctor
  3. 3

    Ask the model for a first reply (the first run downloads the model)

    slotstream run --prompt "Why is the sky blue?"

Things to know

  • Runs only on Apple Silicon Macs with macOS 14 or later; Windows, Linux and Intel Macs are not supported.
  • Needs at least 16 GB of memory and about 110 GB of free SSD space for the default model.
  • The first run downloads a model of about 105 GB, which can take hours.
  • Slotstream maintains only one model, Qwen3.8-Flash-Next 4-bit.

In collections

Claude Code tools, skills & pluginsRun LLMs locallyOpen-source Mac apps

Topics

#apple-silicon#local-llm#mlx#swift#metal#mixture-of-experts#inference-engine#ollama-compatible#claude-code#llm

FAQ

What is slotstream?
A native Swift engine that runs a 105 GB Qwen3.8-Flash-Next 4-bit model on 16–64 GB Apple Silicon Macs by streaming experts from SSD. Slotstream is a local inference engine for Apple Silicon Macs, written in Swift on Apple's MLX and Metal. It runs a mixture-of-experts model whose weights are far larger than system RAM: shared weights stay in memory, while the experts needed for each token are read from SSD into a bounded cache, with frequently used experts kept in RAM. Inference runs entirely on the Mac after the model is downloaded, with no Python runtime or cloud account.
Is slotstream free to use? What license does it have?
slotstream is open source under the MIT license. Check the license terms before using it in commercial products.
Is slotstream still maintained?
Actively maintained. The last commit to slotstream was on Nov 4, 2026.
What can I use slotstream for?
Running a large open-weight model on a 16–64 GB Mac that cannot hold it in RAM. Using Claude Code or Codex with a local model so code stays on your machine. Connecting a local chat front end such as Open WebUI to an offline model server. Embedding a local LLM in a native macOS app written in Swift.
How do I install slotstream?
Run the official install script from the README Check your Mac's memory and disk space: slotstream doctor Ask the model for a first reply (the first run downloads the model): slotstream run --prompt "Why is the sky blue?"
Where can I get slotstream?
The source code and installation instructions are on GitHub: https://github.com/carloslfu/slotstream

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects