Repolode
Submit
EvolvingLMMs-Lab

EvolvingLMMs-Lab/lmms-engine

A unified training engine for multimodal models, supporting vision-language, diffusion, and MoE architectures with FSDP2, Muon, and Liger kernels.

View on GitHubWebsite
Stars
830
Forks
38
Open issues
11
Last commit
15 hours ago

What is it?

LMMs Engine is a Python training framework for multimodal models, aimed at researchers and engineers who want to pretrain or fine-tune vision-language, audio-language, diffusion, and language models from one codebase. Training runs are configured through YAML files and launched with torchrun, Accelerate, or a single-GPU command.

The project covers distributed training with FSDP2, tensor and Ulysses sequence parallelism, and expert parallelism for Mixture-of-Experts models. It also includes memory and speed optimizations such as sequence packing with unpadding, Liger Triton kernels, native sparse attention for BAGEL, and the Muon optimizer. Supported model families include Qwen2.5-VL, Qwen3-VL, Qwen3-Omni MoE, BAGEL, WanVideo, and SiT, and new architectures can be added with a model registration decorator.

The repository provides example run scripts and configs for each supported model, along with dataset preparation and training guides. Docker images with PyTorch, CUDA, and FlashAttention are also published.

Key features

  • 01Distributed training with FSDP2, tensor parallelism, Ulysses sequence parallelism, and expert parallelism for MoE models.
  • 02Sequence packing with first-fit bin packing and unpadded Flash Attention.
  • 03Liger Triton kernels for fused CrossEntropy, RMSNorm, RoPE, and SwiGLU.
  • 04Muon optimizer with Newton-Schulz orthogonalization, combined with Adam for other parameters.
  • 05YAML-driven training configs launched via torchrun, Accelerate, or single GPU.
  • 06Extensible registries for models, datasets, and processors.

Use cases

  • Vision-language pretraining or supervised fine-tuning of Qwen-VL style models on large multimodal datasets.
  • Training Mixture-of-Experts multimodal models with expert parallelism across GPUs.
  • Experimenting with diffusion or flow-matching video and image generation models such as WanVideo or SiT.
  • Pretraining language models with the Muon optimizer and streaming datasets.

Topics

#multimodal#training-framework#pytorch#distributed-training#vision-language-model#diffusion-model#mixture-of-experts#fsdp#agi#large-language-models

FAQ

What is lmms-engine?
A unified training engine for multimodal models, supporting vision-language, diffusion, and MoE architectures with FSDP2, Muon, and Liger kernels. LMMs Engine is a Python training framework for multimodal models, aimed at researchers and engineers who want to pretrain or fine-tune vision-language, audio-language, diffusion, and language models from one codebase. Training runs are configured through YAML files and launched with torchrun, Accelerate, or a single-GPU command.
Is lmms-engine free to use? What license does it have?
lmms-engine has no declared license. Its code is public, but without a license you have no explicit permission to reuse it; ask the author before relying on it.
Is lmms-engine still maintained?
Actively maintained. The last commit to lmms-engine was on Oct 18, 2026.
What can I use lmms-engine for?
Vision-language pretraining or supervised fine-tuning of Qwen-VL style models on large multimodal datasets. Training Mixture-of-Experts multimodal models with expert parallelism across GPUs. Experimenting with diffusion or flow-matching video and image generation models such as WanVideo or SiT. Pretraining language models with the Muon optimizer and streaming datasets.
Where can I get lmms-engine?
The source code and installation instructions are on GitHub: https://github.com/EvolvingLMMs-Lab/lmms-engine

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects