Repolode
Submit
antirez

antirez/ds4

A narrow native inference engine for running DeepSeek V4 Flash and other selected large models on consumer Metal, CUDA and ROCm hardware.

View on GitHub
Stars
23,949
Forks
2,313
Open issues
790
Last commit
7 hours ago

What is it?

DwarfStar is a small, self-contained C inference engine built to run a handful of large open-weight models on hardware people can own. It targets DeepSeek V4 Flash (with an experimental vision model), DeepSeek V4.1 Flash, DeepSeek V4 PRO, GLM 5.x variants, and Qwen3.8 Flash Next. It is deliberately not a general GGUF runner: it uses GGUF files produced by the project itself.

The main target is Apple Silicon Macs, with SSD streaming for machines with less RAM. NVIDIA CUDA is supported on DGX Spark and multi-GPU setups including Ada Lovelace and L40S cards, and ROCm is supported on Strix Halo systems. The project ships an interactive CLI, a native coding agent (ds4-agent) that runs without an HTTP server, an HTTP server (ds4-server) with multi-session support, and an eval tool for capability regression tests.

The project is in beta and changes quickly. Model support is selected based on the best available open weights for consumer-sized machines, and the code is partly developed with AI coding agents.

Key features

  • 01Native C inference engine with Metal, CUDA and ROCm backends.
  • 02SSD streaming lets models larger than RAM run on smaller machines.
  • 03Multi-GPU CUDA support, including Ada Lovelace and L40S cards.
  • 04Built-in native coding agent and OpenAI-style HTTP server with multiple sessions.
  • 05Speculative decoding (MTP) and KV cache save/restore for sessions.

Use cases

  • Running DeepSeek or GLM class models locally on a high-memory Mac.
  • Serving a local multi-user LLM endpoint on a multi-GPU CUDA server.
  • Using a local coding agent with clients such as Pi, OpenCode or Claude Code.
  • Experimenting with inference internals for a small set of large models.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/antirez/ds4.git
  2. 2

    Enter the project directory

    cd ds4
  3. 3

    Build for your platform (Metal on Apple Silicon)

    make
  4. 4

    Download the DeepSeek V4 Flash Q2 model (for 96 or 128 GB machines)

    ./download_model.sh ds4f-q2

Things to know

  • Beta quality and fast changing, so instabilities and regressions are possible.
  • Metal support for DeepSeek and GLM needs 96 GB or more; smaller Macs need SSD streaming.
  • It is deliberately narrow: it runs only the specific supported models and their project-produced GGUF files, not general GGUF files.
  • The software was developed with strong assistance from AI coding agents.

Topics

#llm-inference#deepseek#metal#cuda#rocm#local-llm#gguf#inference-engine

FAQ

What is ds4?
A narrow native inference engine for running DeepSeek V4 Flash and other selected large models on consumer Metal, CUDA and ROCm hardware. DwarfStar is a small, self-contained C inference engine built to run a handful of large open-weight models on hardware people can own. It targets DeepSeek V4 Flash (with an experimental vision model), DeepSeek V4.1 Flash, DeepSeek V4 PRO, GLM 5.x variants, and Qwen3.8 Flash Next. It is deliberately not a general GGUF runner: it uses GGUF files produced by the project itself.
Is ds4 free to use? What license does it have?
ds4 is open source under the MIT license. Check the license terms before using it in commercial products.
Is ds4 still maintained?
Actively maintained. The last commit to ds4 was on Oct 19, 2026.
What can I use ds4 for?
Running DeepSeek or GLM class models locally on a high-memory Mac. Serving a local multi-user LLM endpoint on a multi-GPU CUDA server. Using a local coding agent with clients such as Pi, OpenCode or Claude Code. Experimenting with inference internals for a small set of large models.
How do I install ds4?
Clone the repository: git clone https://github.com/antirez/ds4.git Enter the project directory: cd ds4 Build for your platform (Metal on Apple Silicon): make Download the DeepSeek V4 Flash Q2 model (for 96 or 128 GB machines): ./download_model.sh ds4f-q2
Where can I get ds4?
The source code and installation instructions are on GitHub: https://github.com/antirez/ds4

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects