Repolode
Submit
showlab

showlab/livecc

A video LLM trained with streaming speech transcription that produces real-time commentary on live video, from the CVPR 2025 LiveCC paper.

View on GitHub
Stars
480
Forks
59
Open issues
16
Last commit
12 months ago

What is it?

LiveCC is a video large language model trained on videos paired with streaming speech transcription, so it can generate running commentary while a video plays rather than only answering questions about a finished clip. The project is the research code for the CVPR 2025 paper of the same name, covering training, inference and evaluation.

The repository provides a Gradio web demo and a command-line demo for trying the model, along with pre-training and supervised fine-tuning scripts built on DeepSpeed and Hugging Face Transformers. Published checkpoints, datasets such as Live-WhisperX-526K, and a benchmark called LiveSports3KCC are linked from the README. Evaluation scripts cover VideoMME and OVOBench, and a data production pipeline is included for users who want to rebuild the training data.

The code targets Python 3.11 or newer and is built on Qwen2-VL-based models, with Flash Attention and Liger kernels used for efficiency.

Key features

  • 01Generates real-time video commentary using streaming speech transcription training.
  • 02Includes a Gradio web demo and a command-line demo.
  • 03Provides pre-training and SFT scripts using DeepSpeed.
  • 04Ships evaluation scripts for LiveSports3KCC, VideoMME and OVOBench.
  • 05Links published checkpoints and training datasets on Hugging Face.

Use cases

  • Researchers studying video LLMs trained on speech-aligned data.
  • Teams building live sports or event commentary prototypes.
  • Developers reproducing the LiveCC paper's training and evaluation results.

Quick start

  1. 1

    Install Python 3.11 or newer, then install PyTorch

    pip install torch torchvision torchaudio
  2. 2

    Install the project's Python dependencies and livecc-utils

    pip install "transformers>=4.52.4" accelerate deepspeed peft opencv-python decord datasets tensorboard gradio pillow-heif gpustat timm sentencepiece openai av==12.0.0 qwen_vl_utils liger_kernel numpy==1.24.4 && pip install flash-attn --no-build-isolation && pip install livecc-utils==0.0.2
  3. 3

    Launch the Gradio demo

    python demo/app.py --js_monitor

Things to know

  • The README does not state a license, so usage rights are unclear.
  • Setup needs a CUDA GPU and several heavy dependencies, including flash-attn and deepspeed.
  • Evaluation with the default GPT-4o judge requires an Azure OpenAI account.

Topics

#video-llm#multimodal#speech-transcription#real-time#computer-vision#deep-learning#pytorch

FAQ

What is livecc?
A video LLM trained with streaming speech transcription that produces real-time commentary on live video, from the CVPR 2025 LiveCC paper. LiveCC is a video large language model trained on videos paired with streaming speech transcription, so it can generate running commentary while a video plays rather than only answering questions about a finished clip. The project is the research code for the CVPR 2025 paper of the same name, covering training, inference and evaluation.
Is livecc free to use? What license does it have?
livecc has no declared license. Its code is public, but without a license you have no explicit permission to reuse it; ask the author before relying on it.
Is livecc still maintained?
No commits for 6+ months. The last commit to livecc was on Oct 29, 2025.
What can I use livecc for?
Researchers studying video LLMs trained on speech-aligned data. Teams building live sports or event commentary prototypes. Developers reproducing the LiveCC paper's training and evaluation results.
How do I install livecc?
Install Python 3.11 or newer, then install PyTorch: pip install torch torchvision torchaudio Install the project's Python dependencies and livecc-utils: pip install "transformers>=4.52.4" accelerate deepspeed peft opencv-python decord datasets tensorboard gradio pillow-heif gpustat timm sentencepiece openai av==12.0.0 qwen_vl_utils liger_kernel numpy==1.24.4 && pip install flash-attn --no-build-isolation && pip install livecc-utils==0.0.2 Launch the Gradio demo: python demo/app.py --js_monitor
Where can I get livecc?
The source code and installation instructions are on GitHub: https://github.com/showlab/livecc

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects