Repolode
Submit
huggingface

huggingface/nanotron

A minimalist library from Hugging Face for pretraining transformer LLMs with 3D parallelism (DP, TP, PP) on multi-GPU clusters.

View on GitHub
Stars
2,855
Forks
342
Open issues
159
Last commit
last month

What is it?

Nanotron is a Python library for pretraining transformer language models on custom datasets. It aims to keep the training API simple and readable while scaling to large multi-GPU and multi-node setups.

The library implements data, tensor, and pipeline parallelism, along with expert parallelism for Mixture-of-Experts models. It offers AFAB and 1F1B pipeline schedules, a ZeRO-1 optimizer, FP32 gradient accumulation, and explicit TP and PP APIs intended to make debugging easier. Example configurations cover a tiny Llama model, Mamba, MoE, DoReMi, and spectral µTransfer scaling.

Training is driven by YAML config files launched with torchrun, and checkpoints can be generated from with a separate script. The project also points to an Ultrascale Playbook with benchmark data for different model sizes and node counts.

Key features

  • 013D parallelism combining data, tensor, and pipeline parallelism
  • 02Expert parallelism for Mixture-of-Experts models
  • 03AFAB and 1F1B pipeline schedules
  • 04ZeRO-1 optimizer with FP32 gradient accumulation
  • 05Spectral µTransfer parametrization for scaling models

Use cases

  • Pretraining a custom Llama-style language model on multi-GPU hardware
  • Experimenting with Mixture-of-Experts or Mamba architectures at small scale
  • Scaling transformer training across multiple nodes with Slurm

Quick start

  1. 1

    Create a Python virtual environment with uv

    uv venv nanotron --python 3.11 && source nanotron/bin/activate && uv pip install --upgrade pip
  2. 2

    Install PyTorch

    uv pip install torch --index-url https://download.pytorch.org/whl/cu124
  3. 3

    Install the core dependencies

    uv pip install -e .
  4. 4

    Train a tiny Llama model on a single 8-GPU node

    CUDA_DEVICE_MAX_CONNECTIONS=1 torchrun --nproc_per_node=8 run_train.py --config-file examples/config_tiny_llama.yaml

Things to know

  • Training examples assume multi-GPU NVIDIA hardware, such as an 8 x H100 node.
  • Setup requires several extra steps, including PyTorch CUDA wheels and flash-attn builds.
  • Some features such as FP8 training and ZeRO-3 are still on the roadmap.

Topics

#llm#pretraining#distributed-training#pytorch#model-parallelism#transformer

FAQ

What is nanotron?
A minimalist library from Hugging Face for pretraining transformer LLMs with 3D parallelism (DP, TP, PP) on multi-GPU clusters. Nanotron is a Python library for pretraining transformer language models on custom datasets. It aims to keep the training API simple and readable while scaling to large multi-GPU and multi-node setups.
Is nanotron free to use? What license does it have?
nanotron is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is nanotron still maintained?
Updated within 6 months. The last commit to nanotron was on Oct 8, 2026.
What can I use nanotron for?
Pretraining a custom Llama-style language model on multi-GPU hardware Experimenting with Mixture-of-Experts or Mamba architectures at small scale Scaling transformer training across multiple nodes with Slurm
How do I install nanotron?
Create a Python virtual environment with uv: uv venv nanotron --python 3.11 && source nanotron/bin/activate && uv pip install --upgrade pip Install PyTorch: uv pip install torch --index-url https://download.pytorch.org/whl/cu124 Install the core dependencies: uv pip install -e . Train a tiny Llama model on a single 8-GPU node: CUDA_DEVICE_MAX_CONNECTIONS=1 torchrun --nproc_per_node=8 run_train.py --config-file examples/config_tiny_llama.yaml
Where can I get nanotron?
The source code and installation instructions are on GitHub: https://github.com/huggingface/nanotron

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects