Repolode
Submit
MiaAI-Lab

MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks

Recipe for serving DeepSeek-V4.1-Flash-1017 (EXL3 2.9 bpw) across two NVIDIA DGX Sparks as an OpenAI-compatible API with a 1M-token window.

View on GitHubWebsite
Stars
134
Forks
21
Open issues
7
Last commit
9 hours ago

What is it?

This repository is a deployment recipe that runs the DeepSeek-V4.1-Flash-1017 model split across two NVIDIA DGX Sparks (GB10, 128 GB each) linked by a direct ConnectX-7 cable. The model runs with tensor parallelism of two, one rank per machine, on TensorFold's native Zig engine using a 2.9 bit EXL3 quantized checkpoint. A single script on the first Spark sets up both machines, downloads the weights, and starts both ranks.

The served API is OpenAI-compatible on port 8888 and covers a 1,048,576-token context window, image input, tool calling, and switchable thinking mode. It also uses DSpark speculative decoding with the checkpoint's own draft blocks. Optional settings enable an abliterated variant through a gated overlay, and a memory watchdog stops a rank before it can freeze a machine.

The repository includes benchmark, validation, and output-comparison scripts, along with the measured prefill and decode throughput on one pair of Sparks and notes on memory limits at long contexts.

Key features

  • 01Serves an OpenAI-compatible API on port 8888 with a 1,048,576-token window.
  • 02Splits the model across two DGX Sparks with tensor parallelism of two over a direct ConnectX-7 link.
  • 03Supports image input, tool calling, and thinking on or off per request.
  • 04Uses DSpark speculative decoding from the checkpoint's draft blocks.
  • 05Provides a memory watchdog that stops a rank when free memory runs too low.

Use cases

  • Running a long-context local LLM API on a pair of GB10-class machines.
  • Evaluating a 2.9 bit EXL3 quantized DeepSeek model against its unquantized behavior.
  • Serving an OpenAI-client-compatible endpoint for agents that need tool calling and image input.
  • Studying how tensor-parallel inference is configured across two machines.

Quick start

  1. 1

    Install the Hugging Face CLI on the head Spark

    pip install -U huggingface_hub
  2. 2

    Set up both Sparks and start the ranks from the head Spark

    ./start.sh
  3. 3

    Stop both ranks when finished

    ./stop.sh

Things to know

  • Requires two NVIDIA DGX Sparks (or GB10 systems with 128 GB each) linked by a direct ConnectX-7 cable.
  • The 2.9 bpw quantization is experimental and has no perplexity or KL measurement against the unquantized model.
  • Needs about 196 GiB for the checkpoint plus about 203 GB of extra tables on local disk, and very large memory for long contexts.

Topics

#llm-inference#tensor-parallelism#quantization#openai-compatible-api#deepseek#exllama#speculative-decoding#dgx-spark

FAQ

What is DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks?
Recipe for serving DeepSeek-V4.1-Flash-1017 (EXL3 2.9 bpw) across two NVIDIA DGX Sparks as an OpenAI-compatible API with a 1M-token window. This repository is a deployment recipe that runs the DeepSeek-V4.1-Flash-1017 model split across two NVIDIA DGX Sparks (GB10, 128 GB each) linked by a direct ConnectX-7 cable. The model runs with tensor parallelism of two, one rank per machine, on TensorFold's native Zig engine using a 2.9 bit EXL3 quantized checkpoint. A single script on the first Spark sets up both machines, downloads the weights, and starts both ranks.
Is DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks free to use? What license does it have?
DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks still maintained?
Actively maintained. The last commit to DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks was on Oct 20, 2026.
What can I use DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks for?
Running a long-context local LLM API on a pair of GB10-class machines. Evaluating a 2.9 bit EXL3 quantized DeepSeek model against its unquantized behavior. Serving an OpenAI-client-compatible endpoint for agents that need tool calling and image input. Studying how tensor-parallel inference is configured across two machines.
How do I install DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks?
Install the Hugging Face CLI on the head Spark: pip install -U huggingface_hub Set up both Sparks and start the ranks from the head Spark: ./start.sh Stop both ranks when finished: ./stop.sh
Where can I get DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks?
The source code and installation instructions are on GitHub: https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects