Repolode
Submit
deepspeedai

deepspeedai/DeepSpeed

A deep learning optimization library for making distributed training and inference of large models easier and more efficient.

View on GitHubWebsite
Stars
43,247
Forks
5,048
Open issues
1,540
Last commit
yesterday

What is it?

DeepSpeed is a PyTorch-based library for training and running inference on large deep learning models across many GPUs. It provides system-level techniques such as ZeRO memory partitioning, ZeRO-Infinity offloading to CPU and NVMe, 3D parallelism combining data, tensor and pipeline parallelism, and Ulysses sequence parallelism for long sequences.

The library also includes DeepSpeed-MoE for mixture-of-experts models, compression and quantization tools, and an inference engine. Installation is through pip, with C++/CUDA extensions built just-in-time, and the project lists support for NVIDIA and AMD GPUs as well as Intel Gaudi, Intel XPU, CPU, and Huawei Ascend NPU.

DeepSpeed integrates with Hugging Face Transformers and Accelerate, PyTorch Lightning, MosaicML Composer, and other training frameworks.

Key features

  • 01ZeRO memory optimization that partitions optimizer states, gradients and parameters across GPUs.
  • 02ZeRO-Infinity offloading to CPU memory and NVMe storage.
  • 033D parallelism combining data, tensor and pipeline parallelism.
  • 04Mixture-of-experts training and inference via DeepSpeed-MoE.
  • 05Compression and quantization tools for model size and speed.
  • 06Integrations with Hugging Face Transformers, Accelerate and PyTorch Lightning.

Use cases

  • Training language models with billions of parameters across a multi-GPU cluster.
  • Running inference on models too large to fit in a single GPU's memory.
  • Training with very long input sequences using sequence parallelism.
  • Adding memory offloading to an existing PyTorch training pipeline.

Quick start

  1. 1

    Install PyTorch first (required before DeepSpeed)

  2. 2

    Install DeepSpeed from PyPI

    pip install deepspeed

Things to know

  • Requires PyTorch to be installed before DeepSpeed, and a CUDA or ROCm compiler for its C++/CUDA extensions.
  • Best tested on NVIDIA Pascal–Hopper and AMD MI100/MI200 GPUs; other hardware may be less supported.
  • Large-scale distributed training features are aimed at multi-GPU setups, which adds setup complexity.

Topics

#deep-learning#distributed-training#pytorch#gpu#model-parallelism#inference#llm#mixture-of-experts#billion-parameters#compression

FAQ

What is DeepSpeed?
A deep learning optimization library for making distributed training and inference of large models easier and more efficient. DeepSpeed is a PyTorch-based library for training and running inference on large deep learning models across many GPUs. It provides system-level techniques such as ZeRO memory partitioning, ZeRO-Infinity offloading to CPU and NVMe, 3D parallelism combining data, tensor and pipeline parallelism, and Ulysses sequence parallelism for long sequences.
Is DeepSpeed free to use? What license does it have?
DeepSpeed is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is DeepSpeed still maintained?
Actively maintained. The last commit to DeepSpeed was on Oct 19, 2026.
What can I use DeepSpeed for?
Training language models with billions of parameters across a multi-GPU cluster. Running inference on models too large to fit in a single GPU's memory. Training with very long input sequences using sequence parallelism. Adding memory offloading to an existing PyTorch training pipeline.
How do I install DeepSpeed?
Install PyTorch first (required before DeepSpeed) Install DeepSpeed from PyPI: pip install deepspeed
Where can I get DeepSpeed?
The source code and installation instructions are on GitHub: https://github.com/deepspeedai/DeepSpeed

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects