Repolode
Submit
areal-project

areal-project/AReaL

A large-scale asynchronous reinforcement learning system for training LLM reasoning and agentic models, with RL training for black-box agents via a base_url swap.

View on GitHubWebsite
Stars
5,855
Forks
634
Open issues
136
Last commit
4 hours ago

What is it?

AReaL is a reinforcement learning infrastructure for training LLMs, aimed at reasoning models and agent applications. It uses a fully asynchronous RL training design, so rollout generation and model training run without waiting on each other in lockstep.

The project supports a range of RL algorithms, including GRPO, GSPO, PPO, DAPO, REINFORCE++, RLOO and DPO, with synchronous mode available by setting max_head_offpolicyness to 0. Training backends include Megatron, PyTorch FSDP and PyTorch Archon, and inference runs on vLLM or SGLang. Example tasks cover GSM8K math, multi-turn math, coding agents, tau2 customer service, search agents, and vision-language models.

For agent developers, AReaL offers online RL for black-box agents: an existing agent can be trained by pointing its base_url and api_key at AReaL's RL service, without code changes. Setup uses uv and a flash-attn wheel, and experiments can run on a single node or a Ray cluster.

Key features

  • 01Fully asynchronous RL training for higher throughput and scalability
  • 02Online RL for black-box agents by replacing the base_url
  • 03Supports GRPO, GSPO, PPO, DAPO, DPO and other algorithms
  • 04Training backends: Megatron, PyTorch FSDP, PyTorch Archon
  • 05Inference backends: vLLM and SGLang, with LoRA support

Use cases

  • Training reasoning models on math benchmarks such as GSM8K
  • Improving an existing agent with RL without rewriting its code
  • Training multi-turn coding or customer-service agents on tau2-bench or SWE tasks
  • Running large-scale RL experiments on Ray or Slurm-style GPU clusters

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/areal-project/AReaL
  2. 2

    Install uv

    pip install uv
  3. 3

    Install training packages and SGLang (CUDA)

    uv sync --extra cuda
  4. 4

    Run the GSM8K GRPO example on a single node

    python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local

Things to know

  • Setup assumes an NVIDIA CUDA GPU environment and a large set of heavy dependencies such as flash-attn.
  • Full training and inference require substantial GPU hardware, and the README's flash-attn wheel is specific to Linux x86_64 and certain Python and CUDA versions.

Topics

#reinforcement-learning#llm#agent#llm-reasoning#distributed-training#grpo#vllm#sglang#llm-agent#machine-learning-systems

FAQ

What is AReaL?
A large-scale asynchronous reinforcement learning system for training LLM reasoning and agentic models, with RL training for black-box agents via a base_url swap. AReaL is a reinforcement learning infrastructure for training LLMs, aimed at reasoning models and agent applications. It uses a fully asynchronous RL training design, so rollout generation and model training run without waiting on each other in lockstep.
Is AReaL free to use? What license does it have?
AReaL is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is AReaL still maintained?
Actively maintained. The last commit to AReaL was on Oct 20, 2026.
What can I use AReaL for?
Training reasoning models on math benchmarks such as GSM8K Improving an existing agent with RL without rewriting its code Training multi-turn coding or customer-service agents on tau2-bench or SWE tasks Running large-scale RL experiments on Ray or Slurm-style GPU clusters
How do I install AReaL?
Clone the repository: git clone https://github.com/areal-project/AReaL Install uv: pip install uv Install training packages and SGLang (CUDA): uv sync --extra cuda Run the GSM8K GRPO example on a single node: python3 examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml scheduler.type=local
Where can I get AReaL?
The source code and installation instructions are on GitHub: https://github.com/areal-project/AReaL

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects