fudan-generative-vision/hallo

What is it?

Key features

• 01Generates a talking portrait video from one image and one WAV audio file.
• 02Controls pose, face and lip motion strength with command-line weights.
• 03Includes two-stage training scripts and a data preprocessing pipeline.
• 04Built on Stable Diffusion v1.5 with an AnimateDiff motion module.
• 05Pretrained models are available on Hugging Face.

Use cases

• Researchers studying audio-driven facial animation and talking-head generation.
• Creators who want to make a portrait photo speak a recorded English voiceover.
• Developers training custom portrait animation models on their own video data.

Quick start

1. 1

   Create and activate a conda environment

   conda create -n hallo python=3.10 && conda activate hallo
2. 2

   Install Python packages with pip

   pip install -r requirements.txt && pip install .
3. 3

   Install ffmpeg (required; Ubuntu example)

   apt-get install ffmpeg
4. 4

   Download the pretrained models into the pretrained_models directory, then run inference on the sample image and audio

   python scripts/inference.py --source_image examples/reference_images/1.jpg --driving_audio examples/driving_audios/1.wav

Things to know

• !Requires a CUDA GPU; the README tests only on Ubuntu 20.04/22.04 with CUDA 12.1 and an A100.
• !Requires downloading several gigabytes of pretrained models before inference works.
• !Driving audio must be English WAV, since training data is English only.
• !Source images must be square, front-facing portraits with the face filling 50%-70% of the frame.

In collections

Topics

FAQ

What is hallo?

Open-source research code that animates a portrait photo to speak in sync with a driving audio clip, producing an MP4 video. Hallo generates talking-head videos from a single portrait image and a WAV audio file. The model drives facial expressions, head pose and lip movement from the audio, so a still photo can appear to speak the recording. Inference is run from the command line with scripts/inference.py, and the output is saved as an MP4 file.

Is hallo free to use? What license does it have?

hallo is open source under the MIT license. Check the license terms before using it in commercial products.

Is hallo still maintained?

No commits for 6+ months. The last commit to hallo was on Sep 14, 2024.

What can I use hallo for?

Researchers studying audio-driven facial animation and talking-head generation. Creators who want to make a portrait photo speak a recorded English voiceover. Developers training custom portrait animation models on their own video data.

How do I install hallo?

Create and activate a conda environment: conda create -n hallo python=3.10 && conda activate hallo Install Python packages with pip: pip install -r requirements.txt && pip install . Install ffmpeg (required; Ubuntu example): apt-get install ffmpeg Download the pretrained models into the pretrained_models directory, then run inference on the sample image and audio: python scripts/inference.py --source_image examples/reference_images/1.jpg --driving_audio examples/driving_audios/1.wav

Where can I get hallo?

The source code and installation instructions are on GitHub: https://github.com/fudan-generative-vision/hallo