Repolode
Submit
NVIDIA-NeMo

NVIDIA-NeMo/labs-molt

An agentic-first, PyTorch and HuggingFace-native RL framework for research, combining Ray, vLLM and NVIDIA AutoModel with FSDP2 for async multi-turn agent training.

View on GitHub
Stars
1,219
Forks
124
Open issues
26
Last commit
2 days ago

What is it?

Molt is a reinforcement learning framework for training LLMs and VLMs on agentic tasks. Reward logic is written as ordinary Python inside an Env or ChatAgent subclass, covering graders, multi-turn tool use, vision-language environments and LLM-as-judge scoring. Ray handles placement and async queues, vLLM runs rollouts, and NVIDIA AutoModel with FSDP2 performs training in pure PyTorch.

The framework supports SFT and online RL, with estimators such as REINFORCE, GRPO, RLOO, GAE-based PPO and on-policy distillation. Parallelism includes tensor, expert and context parallelism for MoE models, along with optimizer CPU offload. Rollout and training overlap through an async queue, and token ids, logprobs and multimodal tensors are kept aligned from generation to training.

The codebase is roughly 9.2K lines of RL code and is installed via a CUDA 13 container or a local pip install that pins a specific AutoModel commit.

Key features

  • 01Env and ChatAgent interfaces let reward functions be plain Python, including multi-turn tools and VLM tasks.
  • 02Ray-based async rollout queues overlap vLLM generation with FSDP2 training.
  • 03Supports TP, EP and CP parallelism plus optimizer CPU offload for large MoE models.
  • 04Estimators include REINFORCE, GRPO, RLOO, PPO with GAE and on-policy distillation.
  • 05Token-level alignment keeps ids, logprobs, action ranges and multimodal tensors consistent through training.
  • 06OpenAI and Anthropic compatible endpoints are served from the rollout engine for external agent harnesses.

Use cases

  • Researching RL fine-tuning of LLMs on math or code tasks with custom graders.
  • Training vision-language models with multi-turn Python tool use.
  • Running on-policy distillation from a larger teacher checkpoint to a smaller student.
  • Scaling RL experiments to large MoE models across multiple GPU nodes.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/NVIDIA-NeMo/labs-molt.git
  2. 2

    Enter the project directory

    cd labs-molt
  3. 3

    Install locally with vLLM extras (requires CUDA 13)

    pip install -e ".[vllm]"

Things to know

  • Requires CUDA 13 (torch 2.13.0+cu130); CUDA 12 environments will not work.
  • With a host driver older than 580, you must use the container instead of a local install.
  • The PyPI molt-rl package lags the pinned AutoModel and is not a supported install path.

In collections

Reinforcement learning frameworks & training environments

Topics

#reinforcement-learning#llm#agentic-rl#rlhf#pytorch#vllm#ray#fsdp#moe#context-parallelism

FAQ

What is labs-molt?
An agentic-first, PyTorch and HuggingFace-native RL framework for research, combining Ray, vLLM and NVIDIA AutoModel with FSDP2 for async multi-turn agent training. Molt is a reinforcement learning framework for training LLMs and VLMs on agentic tasks. Reward logic is written as ordinary Python inside an Env or ChatAgent subclass, covering graders, multi-turn tool use, vision-language environments and LLM-as-judge scoring. Ray handles placement and async queues, vLLM runs rollouts, and NVIDIA AutoModel with FSDP2 performs training in pure PyTorch.
Is labs-molt free to use? What license does it have?
labs-molt is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is labs-molt still maintained?
Actively maintained. The last commit to labs-molt was on Nov 6, 2026.
What can I use labs-molt for?
Researching RL fine-tuning of LLMs on math or code tasks with custom graders. Training vision-language models with multi-turn Python tool use. Running on-policy distillation from a larger teacher checkpoint to a smaller student. Scaling RL experiments to large MoE models across multiple GPU nodes.
How do I install labs-molt?
Clone the repository: git clone https://github.com/NVIDIA-NeMo/labs-molt.git Enter the project directory: cd labs-molt Install locally with vLLM extras (requires CUDA 13): pip install -e ".[vllm]"
Where can I get labs-molt?
The source code and installation instructions are on GitHub: https://github.com/NVIDIA-NeMo/labs-molt

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects