Repolode
Submit
baidu-baige

baidu-baige/LoongForge

A training framework for LLMs, VLMs, diffusion and embodied models on NVIDIA GPUs and Kunlun XPUs, built on Megatron-LM and torch-native backends.

View on GitHubWebsite
Stars
694
Forks
58
Open issues
30
Last commit
18 hours ago

What is it?

LoongForge is a training framework from the Baidu AI Cloud Baige team for pre-training, mid-training, SFT and LoRA across LLMs, vision-language models, diffusion models and embodied models such as VLA and world-action models. LLM, VLM and diffusion workloads run on a Megatron backend, while embodied models use a torch-native backend with DDP, ZeRO-1, FSDP and HSDP.

The project ships launch scripts and configs for more than 40 open-source models, along with checkpoint conversion between Megatron and Hugging Face formats and data preprocessing tools. It supports both NVIDIA GPUs and Kunlun XPUs through a plugin design, and includes features such as MoE expert-parallel communication overlap, FP8 training, context parallelism and heterogeneous per-component parallelism for VLMs.

The repository is Python-based, Apache-2.0 licensed, and provides Docker images and documentation for installation and tutorials.

Key features

  • 01Megatron and torch-native backends for LLM, VLM, diffusion and embodied training.
  • 02Ready-to-run launch scripts and configs for 40+ open-source models.
  • 03Runs on NVIDIA GPUs and Kunlun XPUs.
  • 04Supports pre-training, mid-training, SFT and LoRA.
  • 05Includes Megatron ↔ Hugging Face checkpoint conversion and data preprocessing tools.
  • 06Offers MoE communication optimization, FP8 training and context parallelism.

Use cases

  • Pre-training or fine-tuning a large MoE language model on a multi-node GPU cluster.
  • Training a vision-language model with independent parallelism settings for the ViT and LLM parts.
  • Fine-tuning VLA robot policies such as Pi0.5 on LIBERO data.
  • Running large-model training workloads on Kunlun XPU hardware.

Things to know

  • Requires NVIDIA GPUs or Kunlun XPUs, so it is not aimed at other hardware.
  • Source code was only made public in April 2026 and the first tagged release is v0.1.0, so the project is young.

Topics

#llm#training-framework#megatron#vlm#diffusion#embodied-ai#distributed-training#xpu#gpu#kunlun

FAQ

What is LoongForge?
A training framework for LLMs, VLMs, diffusion and embodied models on NVIDIA GPUs and Kunlun XPUs, built on Megatron-LM and torch-native backends. LoongForge is a training framework from the Baidu AI Cloud Baige team for pre-training, mid-training, SFT and LoRA across LLMs, vision-language models, diffusion models and embodied models such as VLA and world-action models. LLM, VLM and diffusion workloads run on a Megatron backend, while embodied models use a torch-native backend with DDP, ZeRO-1, FSDP and HSDP.
Is LoongForge free to use? What license does it have?
LoongForge is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is LoongForge still maintained?
Actively maintained. The last commit to LoongForge was on Oct 19, 2026.
What can I use LoongForge for?
Pre-training or fine-tuning a large MoE language model on a multi-node GPU cluster. Training a vision-language model with independent parallelism settings for the ViT and LLM parts. Fine-tuning VLA robot policies such as Pi0.5 on LIBERO data. Running large-model training workloads on Kunlun XPU hardware.
Where can I get LoongForge?
The source code and installation instructions are on GitHub: https://github.com/baidu-baige/LoongForge

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects