Repolode
Submit
asanchezyali

asanchezyali/talking-avatar-with-ai

A digital human avatar that listens to speech, replies with GPT, speaks with ElevenLabs voice, and lip-syncs using Rhubarb.

View on GitHub
Stars
471
Forks
113
Open issues
3
Last commit
last month

What is it?

This project builds a talking digital human that can both listen and respond. Text or spoken input is turned into a reply by OpenAI's GPT, and Whisper transcribes audio input. The reply is then voiced with Eleven Labs text-to-speech, and Rhubarb Lip Sync produces viseme data so the avatar's mouth matches the audio.

The LLM is prompted to return a structured JSON array of up to three messages, each with text, a facial expression, and an animation name. The backend bundles audio and lip-sync data with each message, so a front-end client can play the speech while the avatar performs the matching expressions and gestures. The repository is a yarn monorepo with a backend in apps/backend and a web client served on a local dev port.

Setup requires an OpenAI key, an Eleven Labs key with voice and model IDs, the Rhubarb Lip-Sync binaries placed in a bin directory, and ffmpeg installed. The README recommends a paid Eleven Labs plan, since the free tier can fail under request volume.

Key features

  • 01Speech-to-text input via OpenAI Whisper.
  • 02Replies generated by GPT with structured expression and animation fields.
  • 03Voice synthesis through the Eleven Labs TTS API.
  • 04Lip-sync viseme data generated by Rhubarb Lip Sync.
  • 05Monorepo layout with a backend and a browser client for local development.

Use cases

  • Prototyping a conversational character for a demo or game.
  • Learning how to chain LLM, speech-to-text, TTS and lip-sync in one pipeline.
  • Building a virtual host or assistant with a animated face.

Quick start

  1. 1

    Clone the repository

  2. 2

    Install dependencies for the monorepo from the project directory

    yarn
  3. 3

    Create a .env file in /apps/backend/ with your OpenAI and Eleven Labs keys

  4. 4

    Run the development system

    yarn dev

Things to know

  • No license is specified, so usage rights are unclear.
  • Requires paid OpenAI and Eleven Labs subscriptions; the free Eleven Labs tier reportedly fails due to request limits.
  • Rhubarb Lip-Sync and ffmpeg must be installed manually before it runs.

Topics

#ai-avatars#digital-human#lip-sync#text-to-speech#speech-to-text#llm#openai#elevenlabs#lipsync#llms

FAQ

What is talking-avatar-with-ai?
A digital human avatar that listens to speech, replies with GPT, speaks with ElevenLabs voice, and lip-syncs using Rhubarb. This project builds a talking digital human that can both listen and respond. Text or spoken input is turned into a reply by OpenAI's GPT, and Whisper transcribes audio input. The reply is then voiced with Eleven Labs text-to-speech, and Rhubarb Lip Sync produces viseme data so the avatar's mouth matches the audio.
Is talking-avatar-with-ai free to use? What license does it have?
talking-avatar-with-ai has no declared license. Its code is public, but without a license you have no explicit permission to reuse it; ask the author before relying on it.
Is talking-avatar-with-ai still maintained?
Updated within 6 months. The last commit to talking-avatar-with-ai was on Sep 16, 2026.
What can I use talking-avatar-with-ai for?
Prototyping a conversational character for a demo or game. Learning how to chain LLM, speech-to-text, TTS and lip-sync in one pipeline. Building a virtual host or assistant with a animated face.
How do I install talking-avatar-with-ai?
Clone the repository Install dependencies for the monorepo from the project directory: yarn Create a .env file in /apps/backend/ with your OpenAI and Eleven Labs keys Run the development system: yarn dev
Where can I get talking-avatar-with-ai?
The source code and installation instructions are on GitHub: https://github.com/asanchezyali/talking-avatar-with-ai

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects