Repolode
Submit
tonyd2wild

tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark

Run-and-benchmark record for serving DeepSeek-V4.1-Flash (552B MoE) across four NVIDIA DGX Sparks using TensorFold and vLLM, with launch scripts and measured results.

View on GitHub
Stars
131
Forks
21
Open issues
0
Last commit
yesterday

What is it?

This repository documents how to serve the 552B-parameter DeepSeek-V4.1-Flash-1017 mixture-of-experts model on a cluster of NVIDIA DGX Spark (GB10) machines. It is organized as "lanes", each a complete setup combining an inference engine, a quantized weight pack, a node count, launch scripts and benchmark numbers. The main lanes use TensorFold's native engine with LibertAI's EXL3 quantized weights across four or two Sparks, with a 1M-token context window; older vLLM lanes are kept with their patches and benchmarks.

Each lane has a run folder containing the scripts, settings, logs and results used to produce the reported figures, such as tokens per second at different concurrency levels, cold prefill speed, and a 69-scenario tool-calling evaluation. An optional uncensored variant applies an abliteration overlay to the attention output weights, and the repository reports its speed and safety-eval differences from the stock lane.

The project is aimed at developers who want to run a very large open-weight model on local multi-node GB10 hardware, and who want reproducible configurations rather than only a model card.

Key features

  • 01Serves DeepSeek-V4.1-Flash-1017 with tensor parallelism across four or two DGX Spark nodes.
  • 02Provides TensorFold and vLLM launch scripts with documented configuration files.
  • 03Includes per-lane benchmark tables for throughput, time to first token and prefill speed.
  • 04Supports a 1,048,576-token context window in the TensorFold TP4 lane.
  • 05Covers chat, tool calling, thinking mode and image input through an OpenAI-compatible endpoint.
  • 06Publishes a 69-scenario tool-use evaluation alongside each lane's results.

Use cases

  • Running a very large open-weight MoE model on a small local cluster of GB10 machines.
  • Reproducing published throughput and latency numbers before adopting a similar setup.
  • Comparing inference engines (TensorFold, vLLM) and quantization formats on identical hardware.
  • Studying how to split a large model across nodes with RDMA networking.

Topics

#llm-inference#deepseek#vllm#tensor-parallelism#quantization#exl3#nvidia-dgx-spark#moe

FAQ

What is DeepSeek-V4.1-Flash-vLLM-DGX-Spark?
Run-and-benchmark record for serving DeepSeek-V4.1-Flash (552B MoE) across four NVIDIA DGX Sparks using TensorFold and vLLM, with launch scripts and measured results. This repository documents how to serve the 552B-parameter DeepSeek-V4.1-Flash-1017 mixture-of-experts model on a cluster of NVIDIA DGX Spark (GB10) machines. It is organized as "lanes", each a complete setup combining an inference engine, a quantized weight pack, a node count, launch scripts and benchmark numbers. The main lanes use TensorFold's native engine with LibertAI's EXL3 quantized weights across four or two Sparks, with a 1M-token context window; older vLLM lanes are kept with their patches and benchmarks.
Is DeepSeek-V4.1-Flash-vLLM-DGX-Spark free to use? What license does it have?
DeepSeek-V4.1-Flash-vLLM-DGX-Spark is open source under the MIT license. Check the license terms before using it in commercial products.
Is DeepSeek-V4.1-Flash-vLLM-DGX-Spark still maintained?
Actively maintained. The last commit to DeepSeek-V4.1-Flash-vLLM-DGX-Spark was on Oct 19, 2026.
What can I use DeepSeek-V4.1-Flash-vLLM-DGX-Spark for?
Running a very large open-weight MoE model on a small local cluster of GB10 machines. Reproducing published throughput and latency numbers before adopting a similar setup. Comparing inference engines (TensorFold, vLLM) and quantization formats on identical hardware. Studying how to split a large model across nodes with RDMA networking.
Where can I get DeepSeek-V4.1-Flash-vLLM-DGX-Spark?
The source code and installation instructions are on GitHub: https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects