Repolode
Submit
MMMU-Benchmark

MMMU-Benchmark/MMMU

Evaluation code and data pointers for the MMMU and MMMU-Pro benchmarks, which test multimodal models on college-level, multi-discipline questions.

View on GitHubWebsite
Stars
598
Forks
56
Open issues
48
Last commit
3 months ago

What is it?

MMMU is a benchmark of about 11.5K multimodal questions drawn from college exams, quizzes and textbooks, spanning six disciplines, 30 subjects and 32 image types such as charts, diagrams, maps and chemical structures. It is meant to test expert-level perception and reasoning rather than general visual recognition.

MMMU-Pro is a harder variant that filters out questions answerable from text alone, adds extra answer options, and includes a vision-only setting where the question is embedded in the image. The repository provides the evaluation code for both benchmarks, with separate folders for each, and the development, validation and test splits.

The test-set answers are now published, so models can be scored locally with the same pipeline used for the validation set. A community-maintained EvalScope integration is also documented for running MMMU against OpenAI-compatible endpoints.

Key features

  • 01Covers 11.5K college-level multimodal questions across six disciplines and 30 subjects.
  • 02Provides MMMU-Pro with filtered questions, extra options and a vision-only input setting.
  • 03Ships development, validation and released test-set answers for local evaluation.
  • 04Documents an EvalScope workflow for OpenAI-compatible model endpoints.

Use cases

  • Benchmarking a vision-language model's expert-level reasoning on academic images.
  • Comparing multimodal models on MMMU-Pro's harder, vision-only setting.
  • Running local test-set evaluation with released answer files.

Quick start

  1. 1

    Install the EvalScope evaluation tool (community-maintained workflow)

  2. 2

    Run MMMU evaluation against an OpenAI-compatible endpoint with EvalScope

    evalscope eval \ --model YOUR_MODEL \ --api-url OPENAI_API_COMPAT_URL \ --api-key EMPTY_TOKEN \ --datasets mmmu \ --limit 10

Things to know

  • This repo is mainly evaluation code and data pointers, not a standalone software product.
  • The quick start uses a third-party community integration, not the official pipeline.
  • Running evaluations needs access to a model endpoint, which may require API keys or compute.

Topics

#multimodal#benchmark#evaluation#large-multimodal-models#visual-question-answering#stem#computer-vision#deep-learning#deep-neural-networks#foundation-models

FAQ

What is MMMU?
Evaluation code and data pointers for the MMMU and MMMU-Pro benchmarks, which test multimodal models on college-level, multi-discipline questions. MMMU is a benchmark of about 11.5K multimodal questions drawn from college exams, quizzes and textbooks, spanning six disciplines, 30 subjects and 32 image types such as charts, diagrams, maps and chemical structures. It is meant to test expert-level perception and reasoning rather than general visual recognition.
Is MMMU free to use? What license does it have?
MMMU is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is MMMU still maintained?
Updated within 6 months. The last commit to MMMU was on Jul 28, 2026.
What can I use MMMU for?
Benchmarking a vision-language model's expert-level reasoning on academic images. Comparing multimodal models on MMMU-Pro's harder, vision-only setting. Running local test-set evaluation with released answer files.
How do I install MMMU?
Install the EvalScope evaluation tool (community-maintained workflow) Run MMMU evaluation against an OpenAI-compatible endpoint with EvalScope: evalscope eval \ --model YOUR_MODEL \ --api-url OPENAI_API_COMPAT_URL \ --api-key EMPTY_TOKEN \ --datasets mmmu \ --limit 10
Where can I get MMMU?
The source code and installation instructions are on GitHub: https://github.com/MMMU-Benchmark/MMMU

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects