A benchmark suite and evaluation toolkit that scores text-to-video and image-to-video generative models across many quality and semantic dimensions.
VBench is a benchmark suite for evaluating video generative models. It breaks the notion of video generation quality into a hierarchy of evaluation dimensions, such as subject and background consistency, temporal flickering, motion smoothness, aesthetic and imaging quality, object class, human action, and spatial relationships. Each dimension comes with a standard prompt suite and an automated evaluation method, so different models can be compared on the same test cases.
The repository is the reference implementation for the VBench series, which also includes VBench++ (image-to-video, long-video evaluation, and trustworthiness checks covering fairness, bias, and safety) and VBench-2.0, which targets intrinsic faithfulness such as commonsense reasoning, physics and human motion. It ships as a pip package with a `vbench` command-line tool, supports multi-GPU evaluation, and can score custom videos for several dimensions.
The project also publishes the sampled videos used in its paper, along with prompt lists and model details, and maintains public leaderboards for comparing video generation models.
Install the vbench package from PyPI (see the PyPI badge in the README)
Follow the README Usage section to run evaluation on your generated videos
Summary generated by AI from the project's README. GitHub stats refresh daily.
Open-source large-scale video generative models for text-to-video, image-to-video, speech-to-video and character animation.
A customizable framework for evaluating LLMs, VLMs and AIGC models on benchmarks, plus inference performance stress testing and result dashboards.
A public leaderboard ranking LLMs by how often they hallucinate when summarizing short documents, scored with Vectara's HHEM evaluation model.
A Python framework that benchmarks LLMs by having them play the Bulls and Cows code-breaking game through LiteLLM-compatible providers.
A PyTorch library of pretrained diffusion models, noise schedulers and building blocks for image, video and audio generation.
A standard Python API for single-agent reinforcement learning environments, plus a set of reference environments, forked from OpenAI's Gym.