Repolode
Submit
EleutherAI

EleutherAI/gpt-neox

A library for training large autoregressive language models on GPUs, built on Megatron-LM and DeepSpeed with model, pipeline and data parallelism.

View on GitHubWebsite
Stars
7,472
Forks
1,126
Open issues
128
Last commit
3 days ago

What is it?

GPT-NeoX is EleutherAI's library for training large-scale autoregressive language models on GPUs. It is built on NVIDIA's Megatron-LM and extends it with techniques from DeepSpeed, including ZeRO and 3D parallelism, and adds its own optimizations. It is aimed at researchers and teams who want to pretrain or finetune models with billions of parameters from scratch.

The library ships with predefined configurations for architectures such as Pythia, PaLM, Falcon and LLaMA, and supports rotary and ALiBi positional embeddings, flash attention, Mixture of Experts via megablocks, and curriculum learning. Jobs can be launched with pdsh, MPI or Slurm, and training can be monitored with Weights & Biases, Comet or TensorBoard. Checkpoints can be exported to Hugging Face Transformers format, and Llama and Mistral weights can be imported.

The project notes that for generic inference, the Hugging Face transformers library is the better choice.

Key features

  • 01Distributed training with ZeRO and 3D parallelism
  • 02Predefined configs for Pythia, PaLM, Falcon and LLaMA architectures
  • 03Flash attention and Mixture of Experts (megablocks) support
  • 04Launch jobs via Slurm, MPI or pdsh on NVIDIA and AMD GPUs
  • 05Export checkpoints to Hugging Face format and import Llama or Mistral weights
  • 06Experiment monitoring with Weights & Biases, Comet and TensorBoard

Use cases

  • Pretraining a custom large language model from scratch on a GPU cluster
  • Reproducing or extending Pythia-style training experiments
  • Finetuning Llama or Mistral checkpoints with multi-node parallelism
  • Running large training jobs on Slurm or HPC supercomputers

Quick start

  1. 1

    Install the basic dependencies from the repository root

    pip install -r requirements/requirements.txt
  2. 2

    Launch a training job with a config file (replace the path with your own config)

    python3 deepy.py train.py /path/to/configs/my_model.yml

Things to know

  • Built for training large models on GPUs, so it needs substantial GPU hardware.
  • The README says it is likely the wrong tool if you only need inference; use Hugging Face transformers instead.
  • Developed and tested mainly on Python 3.8-3.10 and PyTorch 1.8-2.0, and it depends on a DeepSpeed fork (DeeperSpeed) that should be installed in an isolated environment.

In collections

Run and train AI on GPUs

Topics

#llm#distributed-training#deepspeed#megatron#transformers#pretraining#gpu#deepspeed-library#gpt-3#language-model

FAQ

What is gpt-neox?
A library for training large autoregressive language models on GPUs, built on Megatron-LM and DeepSpeed with model, pipeline and data parallelism. GPT-NeoX is EleutherAI's library for training large-scale autoregressive language models on GPUs. It is built on NVIDIA's Megatron-LM and extends it with techniques from DeepSpeed, including ZeRO and 3D parallelism, and adds its own optimizations. It is aimed at researchers and teams who want to pretrain or finetune models with billions of parameters from scratch.
Is gpt-neox free to use? What license does it have?
gpt-neox is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is gpt-neox still maintained?
Actively maintained. The last commit to gpt-neox was on Nov 4, 2026.
What can I use gpt-neox for?
Pretraining a custom large language model from scratch on a GPU cluster Reproducing or extending Pythia-style training experiments Finetuning Llama or Mistral checkpoints with multi-node parallelism Running large training jobs on Slurm or HPC supercomputers
How do I install gpt-neox?
Install the basic dependencies from the repository root: pip install -r requirements/requirements.txt Launch a training job with a config file (replace the path with your own config): python3 deepy.py train.py /path/to/configs/my_model.yml
Where can I get gpt-neox?
The source code and installation instructions are on GitHub: https://github.com/EleutherAI/gpt-neox

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects