Repolode
Submit
NVIDIA-NeMo

NVIDIA-NeMo/Automodel

PyTorch DTensor-native training library for fine-tuning and pretraining LLMs, VLMs and diffusion models directly from Hugging Face checkpoints.

View on GitHubWebsite
Stars
1,078
Forks
369
Open issues
502
Last commit
16 hours ago

What is it?

NeMo AutoModel is an open-source training library in the NVIDIA NeMo Framework, built on PyTorch distributed primitives such as DeviceMesh and DTensor. It targets pretraining and fine-tuning (SFT and PEFT/LoRA) of large language models, vision-language models, diffusion models and retrieval models, and loads Hugging Face checkpoints without format conversion.

Training runs are defined by YAML recipes that can be overridden from the command line. Parallelism is configured rather than hard-coded: the same script can run on a single GPU or many nodes by changing the device mesh, combining FSDP2, HSDP, tensor, context, sequence and pipeline parallelism. Expert parallelism is supported for MoE models, and the project includes launch paths for Kubernetes and Slurm.

The repository ships recipes for a wide range of recent open model families, including Qwen, Gemma, Mistral, GLM, DeepSeek and Nemotron, and adds speculative decoding drafter training such as EAGLE and DFlash.

Key features

  • 01Loads Hugging Face models directly for SFT, LoRA and pretraining without conversion.
  • 02Scales from one GPU to multi-node jobs by changing the device mesh configuration.
  • 03Combines FSDP2, HSDP, tensor, context, sequence and pipeline parallelism.
  • 04Supports expert parallelism for mixture-of-experts models.
  • 05Runs YAML-defined recipes that can be overridden from the CLI.
  • 06Provides Kubernetes and Slurm deployment paths.

Use cases

  • Fine-tuning a large open-weight LLM or VLM on a multi-GPU or multi-node cluster.
  • Running LoRA or full-parameter SFT on recent MoE models with expert parallelism.
  • Training speculative decoding drafter models such as EAGLE for faster inference.
  • Prototyping training changes in plain PyTorch without rewriting model code for each parallel strategy.

Things to know

  • Many example recipes target large multi-node GPU clusters, such as 8 to 16 nodes or EP64 setups.

Topics

#llm#pytorch#fine-tuning#distributed-training#vlm#hugging-face#moe#lora#agent#deepseek-v4

FAQ

What is Automodel?
PyTorch DTensor-native training library for fine-tuning and pretraining LLMs, VLMs and diffusion models directly from Hugging Face checkpoints. NeMo AutoModel is an open-source training library in the NVIDIA NeMo Framework, built on PyTorch distributed primitives such as DeviceMesh and DTensor. It targets pretraining and fine-tuning (SFT and PEFT/LoRA) of large language models, vision-language models, diffusion models and retrieval models, and loads Hugging Face checkpoints without format conversion.
Is Automodel free to use? What license does it have?
Automodel is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is Automodel still maintained?
Actively maintained. The last commit to Automodel was on Nov 6, 2026.
What can I use Automodel for?
Fine-tuning a large open-weight LLM or VLM on a multi-GPU or multi-node cluster. Running LoRA or full-parameter SFT on recent MoE models with expert parallelism. Training speculative decoding drafter models such as EAGLE for faster inference. Prototyping training changes in plain PyTorch without rewriting model code for each parallel strategy.
Where can I get Automodel?
The source code and installation instructions are on GitHub: https://github.com/NVIDIA-NeMo/Automodel

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects