deepseek-ai/DeepInfer

What is it?

Key features

• 01Offline inference for DeepSeek-V4.1-Flash with optimized kernels and a sampler runtime
• 02Attention data parallelism, context parallelism, and MoE expert parallelism with EPLB
• 03DSpark speculative decoding with confidence-scheduled verification
• 04Asynchronous batch scheduling that overlaps CPU work with device execution
• 05Batch-invariant inference and deterministic sampling on NVIDIA GPUs
• 06Structured outputs constrained by regex, JSON schema, or EBNF grammar

Use cases

• Running offline batch inference with DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs
• Generating JSON or other schema-constrained outputs from a large model
• Studying how DeepSeek serves MoE models with expert parallelism and speculative decoding
• Deploying DeepSeek models on Huawei Ascend 950 NPUs

Quick start

1. 1

   Create and activate an empty Python 3.12 environment

2. 2

   Set the accelerator to NVIDIA

   export DEEPINFER_ACCELERATOR=nvidia
3. 3

   Build DeepInfer from source

   ./build.sh
4. 4

   Run offline inference (downloads the default checkpoint on first run)

   python examples/offline_inference.py

Things to know

• !Runs only on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs, with Linux, CUDA 13 or CANN 9.2 required.
• !Supports only the DeepSeek-V4.1-Flash model.
• !This open-source release omits a KV cache system and prefill/decode disaggregation, so it is not a complete serving stack.
• !Some features, such as batch-invariant inference and deterministic sampling, are NVIDIA only.

Topics

FAQ

What is DeepInfer?

An open-source offline inference framework from DeepSeek for running DeepSeek-V4.1-Flash on NVIDIA SM100 and Huawei Ascend 950 hardware. DeepInfer is an inference framework from DeepSeek for serving large DeepSeek models. This release covers offline inference for DeepSeek-V4.1-Flash, with optimized kernels, a model and sampler runtime, and parallelism options including attention data parallelism, context parallelism, and MoE expert parallelism with EPLB.

Is DeepInfer free to use? What license does it have?

DeepInfer is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.

Is DeepInfer still maintained?

Actively maintained. The last commit to DeepInfer was on Oct 20, 2026.

What can I use DeepInfer for?

Running offline batch inference with DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs Generating JSON or other schema-constrained outputs from a large model Studying how DeepSeek serves MoE models with expert parallelism and speculative decoding Deploying DeepSeek models on Huawei Ascend 950 NPUs

How do I install DeepInfer?

Create and activate an empty Python 3.12 environment Set the accelerator to NVIDIA: export DEEPINFER_ACCELERATOR=nvidia Build DeepInfer from source: ./build.sh Run offline inference (downloads the default checkpoint on first run): python examples/offline_inference.py

Where can I get DeepInfer?

The source code and installation instructions are on GitHub: https://github.com/deepseek-ai/DeepInfer