A deep learning optimization library for making distributed training and inference of large models easier and more efficient.
43.2K5KPythonApache-2.0Updated yesterday
NVIDIA/Megatron-LM
GPU-optimized library and reference training scripts for training large transformer models with multi-dimensional parallelism.
- Stars
- 18,209
- Forks
- 4,645
- Open issues
- 1,657
- Last commit
- 7 hours ago
What is it?
Megatron-LM is NVIDIA's framework for training transformer models at scale across many GPUs. The repository contains two parts: Megatron Core, a composable library of GPU-optimized building blocks for writing custom training frameworks, and Megatron-LM, a reference example built on Megatron Core that adds pre-configured training scripts for research and experimentation.
Megatron Core covers tensor, pipeline, data, expert, and context parallelism, along with mixed precision support across FP16, BF16, FP8, and FP4. It includes transformer building blocks and model architectures. Megatron Bridge, a separate project, handles conversion between Hugging Face and Megatron checkpoints.
The project is installable from PyPI as megatron-core or from source, and it ships with documentation covering installation, first training runs, and parallelism strategies.
Key features
- 01Tensor, pipeline, data, expert, and context parallelism for scaling across GPUs.
- 02Mixed precision training with FP16, BF16, FP8, and FP4.
- 03Composable transformer building blocks and model architectures for custom frameworks.
- 04Pre-configured training scripts and examples for research use.
- 05Checkpoint conversion between Hugging Face and Megatron formats via Megatron Bridge.
Use cases
- Training large language models across many GPUs with multi-dimensional parallelism.
- Building a custom distributed training pipeline on top of GPU-optimized building blocks.
- Running research experiments on transformer training with pre-configured scripts.
- Converting checkpoints between Hugging Face and Megatron formats.
Quick start
- 1
Install Megatron Core from PyPI
uv pip install megatron-core - 2
Or clone the repository and install from source
git clone https://github.com/NVIDIA/Megatron-LM.git && cd Megatron-LM && uv pip install -e . - 3
Follow the official Your First Training Run guide for data preparation and training examples
Things to know
- Training large models requires NVIDIA GPUs and substantial cluster resources.
- Building from source can use a lot of memory and may need MAX_JOBS to limit compilation.
- Some features such as the dev branch are experimental.
- The license is listed as NOASSERTION in the metadata, so check the LICENSE file before use.
Topics
FAQ
- What is Megatron-LM?
- GPU-optimized library and reference training scripts for training large transformer models with multi-dimensional parallelism. Megatron-LM is NVIDIA's framework for training transformer models at scale across many GPUs. The repository contains two parts: Megatron Core, a composable library of GPU-optimized building blocks for writing custom training frameworks, and Megatron-LM, a reference example built on Megatron Core that adds pre-configured training scripts for research and experimentation.
- Is Megatron-LM free to use? What license does it have?
- Megatron-LM is open source under the Other license. Check the license terms before using it in commercial products.
- Is Megatron-LM still maintained?
- Actively maintained. The last commit to Megatron-LM was on Oct 20, 2026.
- What can I use Megatron-LM for?
- Training large language models across many GPUs with multi-dimensional parallelism. Building a custom distributed training pipeline on top of GPU-optimized building blocks. Running research experiments on transformer training with pre-configured scripts. Converting checkpoints between Hugging Face and Megatron formats.
- How do I install Megatron-LM?
- Install Megatron Core from PyPI: uv pip install megatron-core Or clone the repository and install from source: git clone https://github.com/NVIDIA/Megatron-LM.git && cd Megatron-LM && uv pip install -e . Follow the official Your First Training Run guide for data preparation and training examples
- Where can I get Megatron-LM?
- The source code and installation instructions are on GitHub: https://github.com/NVIDIA/Megatron-LM
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
A GPU communication library for expert-parallel MoE dispatch and combine, with FP8 dispatch and NVLink/RDMA support for training and inference.
10.3K1.5KCudaMITUpdated 4 days ago
A unified training engine for multimodal models, supporting VLM, diffusion and LLM training with FSDP2, Muon and Liger kernels.
83238PythonNo licenseUpdated 3 days ago
A training framework for LLMs, VLMs, diffusion and embodied models on NVIDIA GPUs and Kunlun XPUs, built on Megatron-LM and torch-native backends.
69458PythonApache-2.0Updated yesterday
An open-source system to run, manage, and scale AI workloads across Kubernetes, Slurm, and 20+ clouds or GPU providers with one interface.
10.7K1.3KPythonApache-2.0Updated 7 hours ago
A free open book covering neural networks and large language models for NLP, from machine learning basics through prompting, alignment and reasoning.
Learning Resources#deep-learning#natural-language-processing#large-language-models#neural-networks#transformers
891153No licenseUpdated 2 days ago