Repolode
Submit
NVIDIA

NVIDIA/cosmos-framework

Training and inference framework for NVIDIA's Cosmos3 omnimodal world models, with distributed training, multiple inference backends, and RL post-training.

View on GitHubWebsite
Stars
619
Forks
204
Open issues
50
Last commit
17 hours ago

What is it?

Cosmos-Framework is NVIDIA's codebase for training and serving the Cosmos3 family of omnimodal world models, which jointly process language, images, video, audio, and action sequences. All code lives in a single Python package, cosmos_framework, with separate entry points for training and inference.

On the training side it provides a distributed trainer using FSDP, tensor, context, and pipeline parallelism, with native distributed checkpoints, HuggingFace safetensors import and export, and dataset adapters for JSONL, WebDataset, and LeRobot formats. Inference supports Diffusers, Transformers, and vLLM backends for offline batch generation and online serving with Ray and Gradio. A server-client pipeline is included for deploying Cosmos3 policies.

The repository also includes a post-training module covering SFT, DPO, GRPO, PPO, on-policy distillation, and DiffusionNFT for diffusion generators, with Hugging Face model integrations such as Qwen and GR00T.

Key features

  • 01Distributed training with FSDP, tensor, context, and pipeline parallelism.
  • 02Native checkpoints with HuggingFace safetensors import and export.
  • 03Dataset adapters for JSONL, WebDataset, and LeRobot formats.
  • 04Offline batch and online serving inference via Diffusers, Transformers, and vLLM backends.
  • 05Post-training algorithms including SFT, DPO, GRPO, PPO, and distillation.

Use cases

  • Fine-tuning Cosmos3 world models on custom video or robot data.
  • Serving Cosmos3 policies for robot control through a server-client pipeline.
  • Running multi-GPU reinforcement learning or distillation on Qwen or GR00T models.

Quick start

  1. 1

    Install system dependencies

    sudo apt-get install -y --no-install-recommends curl ffmpeg git-lfs libgl1 libglib2.0-0 libx11-dev tree wget
  2. 2

    Install the package with uv (CUDA 13.0 recommended)

    uv sync --all-extras --group=cu130-train
  3. 3

    Activate the virtual environment

    source .venv/bin/activate
  4. 4

    Run single-GPU inference with a downloaded Cosmos3-Nano checkpoint

    python -m cosmos_framework.scripts.inference --parallelism-preset=latency -i "inputs/omni/t2v.json" -o outputs/omni_nano --checkpoint-path Cosmos3-Nano --seed=0

Things to know

  • Training recipes are tested on 8x H100 80 GB GPUs, so large GPU hardware is needed.
  • Requires NVIDIA GPUs with CUDA drivers and a specific uv version (0.11.29).
  • Setup involves model weight downloads and CUDA environment configuration, which can be complex.

In collections

Reinforcement learning frameworks & training environmentsRobotics & embodied AI projects

Topics

#world-model#training#inference#distributed-training#diffusion#robotics#reinforcement-learning

FAQ

What is cosmos-framework?
Training and inference framework for NVIDIA's Cosmos3 omnimodal world models, with distributed training, multiple inference backends, and RL post-training. Cosmos-Framework is NVIDIA's codebase for training and serving the Cosmos3 family of omnimodal world models, which jointly process language, images, video, audio, and action sequences. All code lives in a single Python package, cosmos_framework, with separate entry points for training and inference.
Is cosmos-framework free to use? What license does it have?
cosmos-framework is open source under the Other license. Check the license terms before using it in commercial products.
Is cosmos-framework still maintained?
Actively maintained. The last commit to cosmos-framework was on Nov 7, 2026.
What can I use cosmos-framework for?
Fine-tuning Cosmos3 world models on custom video or robot data. Serving Cosmos3 policies for robot control through a server-client pipeline. Running multi-GPU reinforcement learning or distillation on Qwen or GR00T models.
How do I install cosmos-framework?
Install system dependencies: sudo apt-get install -y --no-install-recommends curl ffmpeg git-lfs libgl1 libglib2.0-0 libx11-dev tree wget Install the package with uv (CUDA 13.0 recommended): uv sync --all-extras --group=cu130-train Activate the virtual environment: source .venv/bin/activate Run single-GPU inference with a downloaded Cosmos3-Nano checkpoint: python -m cosmos_framework.scripts.inference --parallelism-preset=latency -i "inputs/omni/t2v.json" -o outputs/omni_nano --checkpoint-path Cosmos3-Nano --seed=0
Where can I get cosmos-framework?
The source code and installation instructions are on GitHub: https://github.com/NVIDIA/cosmos-framework

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects