Repolode
Submit
huggingface

huggingface/alignment-handbook

Training recipes and scripts for continued pretraining and aligning language models with human or AI preferences using SFT, DPO, and ORPO.

View on GitHubWebsite
Stars
5,699
Forks
496
Open issues
100
Last commit
29 days ago

What is it?

The Alignment Handbook is a set of training recipes that cover the post-training pipeline for language models, from continued pretraining through supervised fine-tuning to preference alignment. It targets the gap between instruction tuning and aligning models with human or AI preferences, where public guidance on data, training methods and evaluation has been scarce.

The project is organized as a small set of scripts plus YAML recipe files. Each recipe holds every parameter for one training run, and the repository includes recipes that reproduce Zephyr 7B and examples for SmolLM, SmolLM2 and SmolLM3. Training supports full-weight distributed training with DeepSpeed ZeRO-3, as well as LoRA and QLoRA for parameter-efficient fine-tuning. A gpt2-nl recipe shows how the same pipeline can adapt a model to a new language.

The handbook covers DPO, ORPO, reward modeling and rejection sampling, and it includes instructions for pushing trained models to the Hugging Face Hub. Installation uses uv with PyTorch 2.6.0 and Flash Attention 2.

Key features

  • 01Continued pretraining, SFT, DPO and ORPO training scripts
  • 02YAML recipe files that capture all parameters of a training run
  • 03Full-weight distributed training with DeepSpeed ZeRO-3
  • 04LoRA and QLoRA support for parameter-efficient fine-tuning
  • 05Reproduction recipes for Zephyr 7B and SmolLM model families

Use cases

  • Reproducing the Zephyr 7B alignment pipeline step by step
  • Fine-tuning a small open LLM on your own instruction dataset
  • Adapting a base model to a new language and then applying preference tuning
  • Comparing DPO, KTO and IPO style preference methods with existing recipes

Quick start

  1. 1

    Create a Python virtual environment with uv

    uv venv handbook --python 3.11 && source handbook/bin/activate && uv pip install --upgrade pip
  2. 2

    Install PyTorch v2.6.0

    uv pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu126
  3. 3

    Install the project's remaining dependencies from the repository root

    uv pip install .
  4. 4

    Install Flash Attention 2

    uv pip install "flash-attn==2.7.4.post1" --no-build-isolation

Things to know

  • Requires a CUDA GPU setup and specific PyTorch and Flash Attention versions to reproduce results.
  • Setup involves several manual steps, including a Hugging Face login and installing Git LFS.

Topics

#llm#rlhf#dpo#fine-tuning#transformers#preference-alignment#training-recipes

FAQ

What is alignment-handbook?
Training recipes and scripts for continued pretraining and aligning language models with human or AI preferences using SFT, DPO, and ORPO. The Alignment Handbook is a set of training recipes that cover the post-training pipeline for language models, from continued pretraining through supervised fine-tuning to preference alignment. It targets the gap between instruction tuning and aligning models with human or AI preferences, where public guidance on data, training methods and evaluation has been scarce.
Is alignment-handbook free to use? What license does it have?
alignment-handbook is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is alignment-handbook still maintained?
Actively maintained. The last commit to alignment-handbook was on Sep 23, 2026.
What can I use alignment-handbook for?
Reproducing the Zephyr 7B alignment pipeline step by step Fine-tuning a small open LLM on your own instruction dataset Adapting a base model to a new language and then applying preference tuning Comparing DPO, KTO and IPO style preference methods with existing recipes
How do I install alignment-handbook?
Create a Python virtual environment with uv: uv venv handbook --python 3.11 && source handbook/bin/activate && uv pip install --upgrade pip Install PyTorch v2.6.0: uv pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu126 Install the project's remaining dependencies from the repository root: uv pip install . Install Flash Attention 2: uv pip install "flash-attn==2.7.4.post1" --no-build-isolation
Where can I get alignment-handbook?
The source code and installation instructions are on GitHub: https://github.com/huggingface/alignment-handbook

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects