Repolode
Submit
OpenRLHF

OpenRLHF/OpenRLHF

A Ray- and vLLM-based framework for reinforcement learning from human feedback and agentic RL training of large language and vision-language models.

View on GitHubWebsite
Stars
10,105
Forks
1,049
Open issues
402
Last commit
16 days ago

What is it?

OpenRLHF is a framework for training large language models with reinforcement learning from human feedback (RLHF) and related agentic RL methods. It uses Ray to schedule the actor, reward, reference and critic models across GPUs, and vLLM to handle the sample generation step, which the project notes dominates RLHF training time. Model training is built on DeepSpeed ZeRO-3 and works directly with Hugging Face Transformers checkpoints.

The framework implements PPO, REINFORCE++, REINFORCE++-baseline, GRPO, RLOO, DAPO-style dynamic filtering and several other advantage estimators, selected through command-line flags. It supports single-turn training with custom Python reward functions or remote reward endpoints, and multi-turn agent training where an environment is defined by reset and step methods. Supervised fine-tuning, reward model training and DPO are also included, along with LoRA, MoE, RingAttention and vision-language model training.

Training can run synchronously, asynchronously, or in a hybrid engine mode where models share GPUs. Multi-node jobs can be launched on Ray clusters or SLURM.

Key features

  • 01Ray-based scheduling that places actor, reward, reference and critic models on separate GPUs.
  • 02vLLM-powered sample generation with tensor and pipeline parallelism.
  • 03Supports PPO, REINFORCE++, GRPO, RLOO, and DAPO-style dynamic filtering.
  • 04Single-turn custom reward functions and multi-turn agent environments with reset and step interfaces.
  • 05Supervised fine-tuning, reward model training and DPO in the same codebase.
  • 06Vision-language model RLHF with image inputs in prompts and environment feedback.

Use cases

  • Running PPO or GRPO-style reinforcement learning on a 7B to 70B+ parameter LLM across a GPU cluster.
  • Fine-tuning a model for math or code reasoning with rule-based rewards such as answer verification or test execution.
  • Building multi-turn agents whose training environment returns feedback after each model action.
  • Training a vision-language model with reinforcement learning on image-based tasks.

Things to know

  • Depends on a heavy stack of Ray, vLLM and DeepSpeed, which adds setup effort.
  • Designed for multi-GPU training of large models, so it needs substantial GPU resources.

Topics

#reinforcement-learning#rlhf#llm#ray#vllm#deepspeed#distributed-training#vision-language-model#large-language-models#proximal-policy-optimization

FAQ

What is OpenRLHF?
A Ray- and vLLM-based framework for reinforcement learning from human feedback and agentic RL training of large language and vision-language models. OpenRLHF is a framework for training large language models with reinforcement learning from human feedback (RLHF) and related agentic RL methods. It uses Ray to schedule the actor, reward, reference and critic models across GPUs, and vLLM to handle the sample generation step, which the project notes dominates RLHF training time. Model training is built on DeepSpeed ZeRO-3 and works directly with Hugging Face Transformers checkpoints.
Is OpenRLHF free to use? What license does it have?
OpenRLHF is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is OpenRLHF still maintained?
Actively maintained. The last commit to OpenRLHF was on Oct 5, 2026.
What can I use OpenRLHF for?
Running PPO or GRPO-style reinforcement learning on a 7B to 70B+ parameter LLM across a GPU cluster. Fine-tuning a model for math or code reasoning with rule-based rewards such as answer verification or test execution. Building multi-turn agents whose training environment returns feedback after each model action. Training a vision-language model with reinforcement learning on image-based tasks.
Where can I get OpenRLHF?
The source code and installation instructions are on GitHub: https://github.com/OpenRLHF/OpenRLHF

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects