Repolode
Submit
hpcaitech

hpcaitech/ColossalAI

A PyTorch-based distributed training and inference system for large AI models, offering parallelism, memory optimization and heterogeneous training.

View on GitHubWebsite
Stars
41,432
Forks
4,486
Open issues
479
Last commit
8 hours ago

What is it?

Colossal-AI is a deep learning system aimed at making large model training and inference cheaper and easier to scale. It lets developers write single-device model code and then apply data, pipeline, 1D/2D/2.5D/3D tensor, and sequence parallelism, along with ZeRO and automatic parallelism, through a configuration file and a few lines of code.

The project also covers heterogeneous memory management, such as PatrickStar-style offloading, so models can be trained on hardware with limited GPU memory. Its repository includes example training scripts for models such as LLaMA, GPT, BERT, ViT and MoE, along with inference components and application examples including chat, diffusion and protein structure models.

Installation is available from PyPI or from source, with Linux as the supported operating system and NVIDIA GPUs with CUDA required. Docker images are also provided.

Key features

  • 01Data, pipeline, tensor (1D/2D/2.5D/3D) and sequence parallelism for distributed training
  • 02ZeRO and automatic parallelism strategies
  • 03Heterogeneous memory management that offloads data beyond GPU memory
  • 04Parallelism configured through a configuration file with minimal code changes
  • 05Inference components and example applications for LLMs and diffusion models

Use cases

  • Training large language models across multiple GPUs or nodes
  • Fine-tuning or pretraining large models on limited GPU memory
  • Running LLM inference with optimized memory and throughput
  • Reproducing example training pipelines for LLaMA, GPT or diffusion models

Quick start

  1. 1

    See the Installation section for the PyPI or install-from-source options

Things to know

  • Its features target large models and multi-GPU training, so it offers little benefit without GPUs.

Topics

#deep-learning#distributed-training#model-parallelism#pipeline-parallelism#pytorch#llm#gpu#ai#big-model#data-parallelism

FAQ

What is ColossalAI?
A PyTorch-based distributed training and inference system for large AI models, offering parallelism, memory optimization and heterogeneous training. Colossal-AI is a deep learning system aimed at making large model training and inference cheaper and easier to scale. It lets developers write single-device model code and then apply data, pipeline, 1D/2D/2.5D/3D tensor, and sequence parallelism, along with ZeRO and automatic parallelism, through a configuration file and a few lines of code.
Is ColossalAI free to use? What license does it have?
ColossalAI is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is ColossalAI still maintained?
Actively maintained. The last commit to ColossalAI was on Oct 21, 2026.
What can I use ColossalAI for?
Training large language models across multiple GPUs or nodes Fine-tuning or pretraining large models on limited GPU memory Running LLM inference with optimized memory and throughput Reproducing example training pipelines for LLaMA, GPT or diffusion models
How do I install ColossalAI?
See the Installation section for the PyPI or install-from-source options
Where can I get ColossalAI?
The source code and installation instructions are on GitHub: https://github.com/hpcaitech/ColossalAI

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects