Repolode
Submit
HVision-NKU

HVision-NKU/StoryDiffusion

Open-source Stable Diffusion-based tool for generating character-consistent comic images and long-range image-to-video sequences.

View on GitHub
Stars
6,479
Forks
649
Open issues
119
Last commit
2 years ago

What is it?

StoryDiffusion generates sequences of images that keep the same characters across a story. Its core is a consistent self-attention module that can be plugged into SD1.5 and SDXL-based image diffusion models, so the same character appears consistently across many prompts. The current implementation expects at least three text prompts, with five to six recommended for better layout.

The project also includes a motion predictor for long-range video generation. It predicts motion between user-provided condition images in a compressed image semantic space, and the generated images can be chained into longer videos.

Users can run comic generation through a Jupyter notebook or a local Gradio app. A low-VRAM Gradio version is provided, tested on a 24GB GPU. The video generation model source code and pretrained weights are listed as not yet released.

Key features

  • 01Consistent self-attention keeps characters consistent across a sequence of generated images.
  • 02Hot-pluggable into SD1.5 and SDXL-based image diffusion models.
  • 03Motion predictor generates videos by transitioning between condition images.
  • 04Jupyter notebook and local Gradio demo for comic generation.
  • 05Low-VRAM Gradio version available.

Use cases

  • Creating comic panels featuring the same characters across a story.
  • Producing storyboard-style image sequences with consistent subjects.
  • Extending a set of keyframe images into a short video through the motion predictor.

Topics

#stable-diffusion#image-generation#comic-generation#text-to-image#video-generation#diffusion-models#gradio

FAQ

What is StoryDiffusion?
Open-source Stable Diffusion-based tool for generating character-consistent comic images and long-range image-to-video sequences. StoryDiffusion generates sequences of images that keep the same characters across a story. Its core is a consistent self-attention module that can be plugged into SD1.5 and SDXL-based image diffusion models, so the same character appears consistently across many prompts. The current implementation expects at least three text prompts, with five to six recommended for better layout.
Is StoryDiffusion free to use? What license does it have?
StoryDiffusion is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is StoryDiffusion still maintained?
No commits for 6+ months. The last commit to StoryDiffusion was on Sep 26, 2024.
What can I use StoryDiffusion for?
Creating comic panels featuring the same characters across a story. Producing storyboard-style image sequences with consistent subjects. Extending a set of keyframe images into a short video through the motion predictor.
Where can I get StoryDiffusion?
The source code and installation instructions are on GitHub: https://github.com/HVision-NKU/StoryDiffusion

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects