Repolode
Submit
deepseek-ai

deepseek-ai/DeepEP

A GPU communication library for expert-parallel MoE dispatch and combine, with FP8 dispatch and NVLink/RDMA support for training and inference.

View on GitHub
Stars
10,294
Forks
1,481
Open issues
299
Last commit
4 days ago

What is it?

DeepEP is a communication library for large-scale machine learning training and inference. Its core is expert-parallel (EP) all-to-all GPU kernels for Mixture-of-Experts models, covering token dispatch to experts and combine back to their original ranks, including FP8 dispatch. It offers both high-throughput and low-latency modes behind a single EPBuffer interface, and the communication kernels are compiled at runtime through DeepJIT.

Beyond EP, the project includes experimental primitives for pipeline parallelism, context and data parallelism through batched all-gather, reduce-scatter and all-reduce, and remote memory access for Engram storage. It also provides helpers for dynamic redundant experts, such as NVLink weight prefetch and gradient reduction within an NVLink domain.

The library targets Linux with NVIDIA Hopper or newer GPUs, PyTorch with CUDA, and NCCL. NVLink is used for intranode traffic and RDMA networks such as InfiniBand or RoCE for internode traffic.

Key features

  • 01Expert-parallel all-to-all dispatch and combine kernels for MoE models, with FP8 dispatch support.
  • 02Single EPBuffer interface covering both high-throughput and low-latency modes.
  • 03Communication kernels compiled at runtime by DeepJIT for the current GPU.
  • 04NVLink weight prefetch and gradient reduction for dynamic redundant experts.
  • 05Experimental batched collectives and pipeline-parallel send/recv over RDMA.

Use cases

  • Training large Mixture-of-Experts models across multiple GPUs and nodes.
  • Serving MoE models with prefill and decoding phases that need fast expert routing.
  • Building custom expert load balancing that replicates hot experts across an NVLink domain.

Quick start

  1. 1

    Install the NCCL dependency

    python -m pip install "nvidia-nccl-cu13>=2.32.3" --no-deps
  2. 2

    Initialize the DeepJIT submodule

    git submodule update --init --recursive
  3. 3

    Build a wheel and install it into the current Python environment

    bash install.sh
  4. 4

    Import deep_ep in your Python project

Things to know

  • Requires Linux and NVIDIA Hopper or newer GPUs.
  • Multi-node communication needs an RDMA network, and intranode use needs NVLink.
  • Bucket, Engram, and PP features are experimental.

Topics

#moe#expert-parallelism#gpu#cuda#nccl#rdma#distributed-training#llm-inference

FAQ

What is DeepEP?
A GPU communication library for expert-parallel MoE dispatch and combine, with FP8 dispatch and NVLink/RDMA support for training and inference. DeepEP is a communication library for large-scale machine learning training and inference. Its core is expert-parallel (EP) all-to-all GPU kernels for Mixture-of-Experts models, covering token dispatch to experts and combine back to their original ranks, including FP8 dispatch. It offers both high-throughput and low-latency modes behind a single EPBuffer interface, and the communication kernels are compiled at runtime through DeepJIT.
Is DeepEP free to use? What license does it have?
DeepEP is open source under the MIT license. Check the license terms before using it in commercial products.
Is DeepEP still maintained?
Actively maintained. The last commit to DeepEP was on Oct 16, 2026.
What can I use DeepEP for?
Training large Mixture-of-Experts models across multiple GPUs and nodes. Serving MoE models with prefill and decoding phases that need fast expert routing. Building custom expert load balancing that replicates hot experts across an NVLink domain.
How do I install DeepEP?
Install the NCCL dependency: python -m pip install "nvidia-nccl-cu13>=2.32.3" --no-deps Initialize the DeepJIT submodule: git submodule update --init --recursive Build a wheel and install it into the current Python environment: bash install.sh Import deep_ep in your Python project
Where can I get DeepEP?
The source code and installation instructions are on GitHub: https://github.com/deepseek-ai/DeepEP

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects