Repolode
Submit
wici-ai

wici-ai/inferssd

Runs large mixture-of-experts GGUF models by streaming expert weights from SSD, built on llama.cpp, for machines with less RAM or VRAM than the model needs.

View on GitHubWebsite
Stars
355
Forks
55
Open issues
8
Last commit
6 hours ago

What is it?

InferSSD is an inference engine for large mixture-of-experts (MoE) language models that exceed the RAM or VRAM of the machine running them. It builds on llama.cpp, loads standard GGUF files, and streams expert weights from SSD as needed instead of keeping the whole model in memory.

The project provides a terminal chat mode (`inferssd cli`) and a server mode (`inferssd serve`) that hosts a browser chat UI and an OpenAI-compatible API. It currently supports DeepSeek-V4.1-Flash-1017 and Muse-Spark-Open, and runs on Apple Silicon macOS with Metal and on Linux with NVIDIA CUDA. Windows support is listed as in progress.

Installation is done from source with a single shell script that builds the engine for the local CPU and GPU, taking a few minutes. The first run downloads the model files, roughly 470 GiB for DeepSeek, to a directory of your choice.

Key features

  • 01Runs MoE models larger than available RAM or VRAM by streaming experts from SSD.
  • 02Loads standard GGUF model files built on llama.cpp.
  • 03Provides a terminal chat mode and a browser web UI.
  • 04Exposes an OpenAI-compatible API at a local endpoint.
  • 05Supports Apple Silicon (Metal) and Linux with NVIDIA CUDA.

Use cases

  • Running a large MoE model on a laptop or workstation with limited memory.
  • Serving a local OpenAI-compatible endpoint for tools that expect the OpenAI API.
  • Experimenting with SSD-backed inference on consumer GPUs or Apple Silicon machines.

Quick start

  1. 1

    Run the official install script from the README (builds from source, takes 2 to 12 minutes)

  2. 2

    Chat with a model in your terminal

    inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF
  3. 3

    Or start the web UI and OpenAI-compatible API at http://127.0.0.1:8080

    inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

Things to know

  • Supported only on Apple Silicon macOS and Linux with an NVIDIA GPU; Windows support is still in progress.
  • No prebuilt packages yet, so it is compiled on your machine during installation.
  • The first run downloads about 470 GiB of model files.
  • Linux also needs the CUDA toolkit, liburing headers, and systemd with cgroup v2.

In collections

Run LLMs locallyOpen-source Mac apps

Topics

#llm#inference#mixture-of-experts#llama-cpp#gguf#ssd-offloading#metal#cuda#apple-silicon#ssd

FAQ

What is inferssd?
Runs large mixture-of-experts GGUF models by streaming expert weights from SSD, built on llama.cpp, for machines with less RAM or VRAM than the model needs. InferSSD is an inference engine for large mixture-of-experts (MoE) language models that exceed the RAM or VRAM of the machine running them. It builds on llama.cpp, loads standard GGUF files, and streams expert weights from SSD as needed instead of keeping the whole model in memory.
Is inferssd free to use? What license does it have?
inferssd is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is inferssd still maintained?
Actively maintained. The last commit to inferssd was on Nov 7, 2026.
What can I use inferssd for?
Running a large MoE model on a laptop or workstation with limited memory. Serving a local OpenAI-compatible endpoint for tools that expect the OpenAI API. Experimenting with SSD-backed inference on consumer GPUs or Apple Silicon machines.
How do I install inferssd?
Run the official install script from the README (builds from source, takes 2 to 12 minutes) Chat with a model in your terminal: inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF Or start the web UI and OpenAI-compatible API at http://127.0.0.1:8080: inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF
Where can I get inferssd?
The source code and installation instructions are on GitHub: https://github.com/wici-ai/inferssd

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects