Repolode
Submit
modelscope

modelscope/DiffSynth-Studio

An open-source Diffusion model engine for image, video, and audio generation, with low-VRAM inference and training support.

View on GitHub
Stars
13,276
Forks
1,318
Open issues
566
Last commit
5 hours ago

What is it?

DiffSynth-Studio is a Python framework for running and training Diffusion models, developed by the ModelScope Community. It integrates a range of mainstream open-source models covering image generation and editing, video generation, audio and music generation, and image quality metrics.

Its main focus is making large models usable on modest hardware. Model parameters are scheduled across disk, system memory and VRAM so that consumer GPUs with limited memory can run inference. Quantization to formats such as NF4 and INT8 further reduces memory needs for inference and LoRA training. Most models that support inference also support training, including base models, LoRAs and adapters with extra inputs.

The project also includes a Split Training mode that separates data processing (such as text and VAE encoding) from the gradient-based training stage, along with CPU offload training and differential LoRA training. Related projects built on it include DiffSynth-ComfyUI for ComfyUI workflows and DiffSynth-WebUI for LoRA training.

Key features

  • 01Integrates mainstream open-source models for image, video, audio and music generation.
  • 02Layer-wise scheduling across disk, memory and VRAM for running large models on low-VRAM GPUs.
  • 03Quantization to NF4, INT8 and other formats to reduce inference and LoRA training memory.
  • 04Training support for base models, LoRAs and adapter models with additional inputs.
  • 05Split Training separates non-gradient computations from the training stage to speed up training.
  • 06CPU offload training for LoRA training on consumer GPUs.

Use cases

  • Running large image or video generation models on a consumer GPU with limited VRAM.
  • Training LoRA adapters for Diffusion models on a single workstation.
  • Experimenting with ControlNet, editing or video generation models within one unified Python framework.
  • Building custom inference or training pipelines for newly released open-source Diffusion models.

Things to know

  • The README says development is limited to a few maintainers, so new features and issue responses may be slow.
  • Some old features are no longer maintained after major version updates.

Topics

#diffusion-models#image-generation#video-generation#lora#quantization#pytorch#text-to-image#low-vram

FAQ

What is DiffSynth-Studio?
An open-source Diffusion model engine for image, video, and audio generation, with low-VRAM inference and training support. DiffSynth-Studio is a Python framework for running and training Diffusion models, developed by the ModelScope Community. It integrates a range of mainstream open-source models covering image generation and editing, video generation, audio and music generation, and image quality metrics.
Is DiffSynth-Studio free to use? What license does it have?
DiffSynth-Studio is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is DiffSynth-Studio still maintained?
Actively maintained. The last commit to DiffSynth-Studio was on Oct 20, 2026.
What can I use DiffSynth-Studio for?
Running large image or video generation models on a consumer GPU with limited VRAM. Training LoRA adapters for Diffusion models on a single workstation. Experimenting with ControlNet, editing or video generation models within one unified Python framework. Building custom inference or training pipelines for newly released open-source Diffusion models.
Where can I get DiffSynth-Studio?
The source code and installation instructions are on GitHub: https://github.com/modelscope/DiffSynth-Studio

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects