Repolode
Submit
huggingface

huggingface/trl

A Hugging Face library for post-training transformer language models with SFT, GRPO, DPO, KTO and reward modeling.

View on GitHubWebsite
Stars
19,561
Forks
3,086
Open issues
289
Last commit
4 hours ago

What is it?

TRL is a library for post-training foundation models, covering methods such as Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), Kahneman-Tversky Optimization (KTO) and reward model training. Each method is exposed as a trainer class, such as SFTTrainer, GRPOTrainer, DPOTrainer, KTOTrainer and RewardTrainer, that wraps the Hugging Face Transformers trainer.

TRL builds on the Transformers ecosystem and uses Accelerate to scale from a single GPU to multi-node clusters with DDP, DeepSpeed ZeRO and FSDP. It integrates with PEFT for quantization and LoRA/QLoRA training on modest hardware, and with Unsloth for optimized kernels. A command-line interface lets you run SFT, DPO and KTO without writing code.

The project also ships an experimental area under trl.experimental for features that may change without notice. Recent updates add support for training on AMD Instinct GPUs via ROCm and AWS Trainium via TorchNeuron.

Key features

  • 01Trainer classes for SFT, GRPO, DPO, KTO and reward modeling.
  • 02Multi-GPU and multi-node training via Accelerate, DeepSpeed and FSDP.
  • 03LoRA and QLoRA support through PEFT for training with limited memory.
  • 04Command-line interface for running SFT, DPO and KTO without code.
  • 05Integration with Hugging Face Transformers and the Hub datasets.

Use cases

  • Fine-tuning an open-weight LLM on instruction or chat datasets with SFT.
  • Aligning a model with human preference pairs using DPO.
  • Training a model with reinforcement learning rewards via GRPO.
  • Training a reward model to score responses for downstream RL pipelines.

Quick start

  1. 1

    Install the library from PyPI

    pip install trl
  2. 2

    Run a basic SFT training example from the Python API

  3. 3

    Or fine-tune from the command line with the trl CLI

    trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara --output_dir Qwen2.5-0.5B-SFT

Things to know

  • Training large models needs substantial GPU hardware, and the README points to PEFT/LoRA and quantization to reduce memory needs.
  • The experimental area under trl.experimental may change or be removed without notice.

Topics

#llm#reinforcement-learning#fine-tuning#transformers#rlhf#dpo#peft#pytorch

FAQ

What is trl?
A Hugging Face library for post-training transformer language models with SFT, GRPO, DPO, KTO and reward modeling. TRL is a library for post-training foundation models, covering methods such as Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), Kahneman-Tversky Optimization (KTO) and reward model training. Each method is exposed as a trainer class, such as SFTTrainer, GRPOTrainer, DPOTrainer, KTOTrainer and RewardTrainer, that wraps the Hugging Face Transformers trainer.
Is trl free to use? What license does it have?
trl is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is trl still maintained?
Actively maintained. The last commit to trl was on Oct 21, 2026.
What can I use trl for?
Fine-tuning an open-weight LLM on instruction or chat datasets with SFT. Aligning a model with human preference pairs using DPO. Training a model with reinforcement learning rewards via GRPO. Training a reward model to score responses for downstream RL pipelines.
How do I install trl?
Install the library from PyPI: pip install trl Run a basic SFT training example from the Python API Or fine-tune from the command line with the trl CLI: trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara --output_dir Qwen2.5-0.5B-SFT
Where can I get trl?
The source code and installation instructions are on GitHub: https://github.com/huggingface/trl

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects