Repolode
Submit
microsoft

microsoft/Tutel

An optimized Mixture-of-Experts library from Microsoft for training and serving MoE models in PyTorch, with low-precision inference for models such as DeepSeek, Qwen and GLM.

View on GitHub
Stars
1,029
Forks
111
Open issues
55
Last commit
3 days ago

What is it?

Tutel is a Mixture-of-Experts (MoE) library built on PyTorch. It provides an optimized MoE layer that handles expert routing (gating), data dispatch and all-to-all communication across GPUs, and it lets parallelism, sparsity and capacity settings be switched at runtime without restarting training.

The project also includes an inference stack for large MoE checkpoints. The README describes direct serving of NVFP4, MXFP4 and blockwise FP8 weights for models such as DeepSeek, Kimi, GLM, Qwen3 and GPT-OSS on NVIDIA A100/H100 and AMD MI300 GPUs, packaged as Docker images with an OpenAI-compatible endpoint and an Open-WebUI front end.

Recent releases add fast MoE gating kernels for Qwen3 and DeepSeek V3/R1, NCCL all-to-all variants for variable-length transfers, and a Megablocks path for single-GPU decoding.

Key features

  • 01Dynamic switching of parallelism, capacity and top-k sparsity at runtime
  • 02Optimized MoE layer with built-in FFN experts and custom gate/expert support
  • 03All-to-all communication with optional 2DH mode for large-scale runs
  • 04Low-precision inference for NVFP4, MXFP4 and blockwise FP8 MoE checkpoints
  • 05OpenAI-compatible serving in Docker images for NVIDIA and AMD GPUs

Use cases

  • Building or training MoE layers in a PyTorch model across multiple GPUs
  • Serving large open-weight MoE models such as DeepSeek or Qwen3 on A100, H100 or MI300 hardware
  • Benchmarking all-to-all bandwidth and MoE throughput on a GPU cluster

In collections

Run and train AI on GPUs

Topics

#mixture-of-experts#moe#pytorch#llm#deep-learning#distributed-training#inference#gpu#deepseek

FAQ

What is Tutel?
An optimized Mixture-of-Experts library from Microsoft for training and serving MoE models in PyTorch, with low-precision inference for models such as DeepSeek, Qwen and GLM. Tutel is a Mixture-of-Experts (MoE) library built on PyTorch. It provides an optimized MoE layer that handles expert routing (gating), data dispatch and all-to-all communication across GPUs, and it lets parallelism, sparsity and capacity settings be switched at runtime without restarting training.
Is Tutel free to use? What license does it have?
Tutel is open source under the MIT license. Check the license terms before using it in commercial products.
Is Tutel still maintained?
Actively maintained. The last commit to Tutel was on Nov 4, 2026.
What can I use Tutel for?
Building or training MoE layers in a PyTorch model across multiple GPUs Serving large open-weight MoE models such as DeepSeek or Qwen3 on A100, H100 or MI300 hardware Benchmarking all-to-all bandwidth and MoE throughput on a GPU cluster
Where can I get Tutel?
The source code and installation instructions are on GitHub: https://github.com/microsoft/Tutel

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects