Repolode
Submit
OpenTalker

OpenTalker/SadTalker

Generates a talking-head video from a single portrait image and an audio clip, driven by 3D motion coefficients (CVPR 2023).

Open-source alternative to:HeyGenD-ID

View on GitHubWebsite
Stars
14,183
Forks
2,701
Open issues
671
Last commit
2 years ago

What is it?

SadTalker animates a single portrait photo so that it speaks in sync with a supplied audio file. The project predicts 3D motion coefficients from the audio, covering head pose and facial expression, and uses them to render a talking-face video from the still image.

The repository provides a Gradio-based WebUI that runs locally, a command-line script (inference.py) for batch generation, and a stable-diffusion-webui extension. It supports a full-body still mode, a reference video for motion, and an optional GFPGAN enhancer to improve face quality in the output. Pre-trained checkpoints are distributed through Google Drive, GitHub Releases and Baidu Cloud, and the README describes Linux, Windows and macOS installation paths.

Key features

  • 01Generates talking-face video from one portrait image plus an audio file.
  • 02Predicts head pose and expression from audio via 3D motion coefficients.
  • 03Offers a local Gradio WebUI and a CLI for generation.
  • 04Includes a stable-diffusion-webui extension.
  • 05Supports full-body still mode and an optional GFPGAN face enhancer.

Use cases

  • Creating talking-head videos from a single photo for content or demos.
  • Experimenting with audio-driven facial animation in research.
  • Adding lip-synced avatar speech to a stable-diffusion-webui workflow.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/OpenTalker/SadTalker.git
  2. 2

    Create and activate a Python 3.8 conda environment

    conda create -n sadtalker python=3.8
  3. 3

    Install the requirements

    pip install -r requirements.txt
  4. 4

    Download the pre-trained models (run from the repository root)

    bash scripts/download_models.sh

Things to know

  • Needs a GPU-oriented PyTorch setup with CUDA and a manual model download before first use.
  • License is listed as NOASSERTION in metadata, so check the repository license before commercial use.
  • The README requires a separate Coqui TTS install for the Gradio demo, and Windows and macOS setup have extra steps.

In collections

AI video generationAI video & audio toolsComputer vision libraries & tools

Topics

#talking-face#image-animation#audio-driven#computer-vision#gradio#deep-learning#video-generation#audio-driven-talking-face#cvpr2023#deep-fake

FAQ

What is SadTalker?
Generates a talking-head video from a single portrait image and an audio clip, driven by 3D motion coefficients (CVPR 2023). SadTalker animates a single portrait photo so that it speaks in sync with a supplied audio file. The project predicts 3D motion coefficients from the audio, covering head pose and facial expression, and uses them to render a talking-face video from the still image.
Is SadTalker free to use? What license does it have?
SadTalker is open source under the Other license. Check the license terms before using it in commercial products.
Is SadTalker still maintained?
No commits for 6+ months. The last commit to SadTalker was on Jun 26, 2024.
What can I use SadTalker for?
Creating talking-head videos from a single photo for content or demos. Experimenting with audio-driven facial animation in research. Adding lip-synced avatar speech to a stable-diffusion-webui workflow.
How do I install SadTalker?
Clone the repository: git clone https://github.com/OpenTalker/SadTalker.git Create and activate a Python 3.8 conda environment: conda create -n sadtalker python=3.8 Install the requirements: pip install -r requirements.txt Download the pre-trained models (run from the repository root): bash scripts/download_models.sh
Where can I get SadTalker?
The source code and installation instructions are on GitHub: https://github.com/OpenTalker/SadTalker

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects