Repolode
Submit
open-compass

open-compass/VLMEvalKit

An open-source toolkit for running one-command evaluations of large vision-language models across many multimodal benchmarks.

View on GitHubWebsite
Stars
4,475
Forks
789
Open issues
322
Last commit
yesterday

What is it?

VLMEvalKit (Python package name vlmeval) evaluates large vision-language models (LVLMs) on a range of image and video benchmarks. Users run a single command to evaluate a supported model on a supported benchmark, and the toolkit handles dataset downloading, preprocessing, inference and metric calculation, so there is no need to prepare data across multiple repositories.

All models are evaluated with generation-based evaluation, and results are reported both with exact matching and with LLM-based answer extraction. The project supports a large set of models, including commercial APIs such as GPT-4V and Gemini and open-source models such as Qwen-VL, LLaVA and InternVL. Model developers can add their own model by implementing a single generate_inner() function.

The project publishes official leaderboards on Hugging Face and OpenCompass, and its evaluation records can be downloaded. Setup notes list specific transformers and torchvision versions for individual models.

Key features

  • 01Runs one-command evaluation of LVLMs on dozens of image and video benchmarks.
  • 02Handles data downloading, preprocessing, inference and metric calculation automatically.
  • 03Reports both exact-match and LLM-based answer extraction results.
  • 04Supports commercial APIs and a wide range of open-source vision-language models.
  • 05Lets developers add a new model by implementing a single generate_inner() function.
  • 06Supports multi-node distributed inference via LMDeploy or vLLM.

Use cases

  • Comparing several vision-language models on the same set of benchmarks.
  • Evaluating a self-developed multimodal model against established benchmarks.
  • Reproducing published leaderboard results for multimodal models.
  • Running large-scale evaluations of thinking models with distributed inference.

Quick start

  1. 1

    Read the QuickStart guide for installation and setup

  2. 2

    Run the demo with a supported VLM from Python

Things to know

  • Some VLMs require specific transformers versions, so setup may need version pinning.
  • Results may differ from numbers reported in original papers because it uses generation-based evaluation.

In collections

Claude Code tools, skills & pluginsComputer vision libraries & tools

Topics

#evaluation#vision-language-model#multimodal#benchmark#llm#computer-vision#vqa#pytorch#chatgpt#claude

FAQ

What is VLMEvalKit?
An open-source toolkit for running one-command evaluations of large vision-language models across many multimodal benchmarks. VLMEvalKit (Python package name vlmeval) evaluates large vision-language models (LVLMs) on a range of image and video benchmarks. Users run a single command to evaluate a supported model on a supported benchmark, and the toolkit handles dataset downloading, preprocessing, inference and metric calculation, so there is no need to prepare data across multiple repositories.
Is VLMEvalKit free to use? What license does it have?
VLMEvalKit is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is VLMEvalKit still maintained?
Actively maintained. The last commit to VLMEvalKit was on Nov 6, 2026.
What can I use VLMEvalKit for?
Comparing several vision-language models on the same set of benchmarks. Evaluating a self-developed multimodal model against established benchmarks. Reproducing published leaderboard results for multimodal models. Running large-scale evaluations of thinking models with distributed inference.
How do I install VLMEvalKit?
Read the QuickStart guide for installation and setup Run the demo with a supported VLM from Python
Where can I get VLMEvalKit?
The source code and installation instructions are on GitHub: https://github.com/open-compass/VLMEvalKit

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects