NEO is a family of native vision-language models that process pixels and words in one encoder-free dense architecture, with training and evaluation code.
920+10 this week33PythonApache-2.0Updated 16 hours ago
EvolvingLMMs-Lab/lmms-eval
A unified toolkit for evaluating multimodal models across image, video, and audio tasks, with 100+ benchmarks and 30+ model backends.
- Stars
- 4,488
- Forks
- 686
- Open issues
- 67
- Last commit
- 23 hours ago
What is it?
lmms-eval is an evaluation toolkit for large multimodal models. It runs a model against a shared set of benchmarks covering image, video, and audio understanding, so teams can compare results on the same tasks with the same pipeline instead of stitching together scattered datasets and post-processing scripts.
The project ships with more than 100 tasks, such as MMMU, MME, MMBench, MathVista, VideoMME, and EgoSchema, and supports more than 30 model families including Qwen2.5-VL, Qwen3-VL, LLaVA-OneVision, and InternVL-2. Models can run locally, through vLLM or SGLang servers, or via any OpenAI-compatible API. Users can also add custom benchmarks with YAML configuration and a document-to-messages function.
Beyond the command-line runner, the project includes a web UI for configuring and monitoring runs and an HTTP evaluation server that accepts asynchronous evaluation jobs, which suits evaluating checkpoints during training. It is a fork of lm-evaluation-harness adapted for multimodal inputs.
Key features
- 01Runs 100+ image, video, and audio benchmarks through one pipeline.
- 02Supports 30+ model families, plus OpenAI-compatible API endpoints.
- 03Integrates with vLLM and SGLang for serving-based evaluation.
- 04Provides a web UI for configuring runs and viewing saved results.
- 05Offers an HTTP evaluation server with an async job queue.
- 06Lets users add custom tasks via YAML and a message-building function.
Use cases
- Comparing a multimodal model against standard image or video benchmarks.
- Evaluating training checkpoints automatically during model training.
- Benchmarking API-hosted models such as GPT-4o alongside open-weight models.
- Adding an in-house evaluation dataset to a shared, reproducible pipeline.
Quick start
- 1
Clone the repository
git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git - 2
Install the package in editable mode with uv
cd lmms-eval && uv pip install -e ".[all]" - 3
Run a quick evaluation (Qwen2.5-VL on MME, 8 samples)
python -m lmms_eval --model qwen2_5_vl --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct --tasks mme --batch_size 1 --limit 8
Things to know
- The metadata gives no clear license (NOASSERTION), so check the license terms before reuse.
- Some tasks need API keys, such as OpenAI or Reka, and some caption tasks need Java 8 installed.
- Running evaluations on local models will generally require a GPU and a large set of dependencies.
In collections
Topics
#multimodal#vision-language-model#benchmark#evaluation#video-understanding#audio-evaluation#llm-evaluation#agi#large-language-models#multimodal-evaluation
FAQ
- What is lmms-eval?
- A unified toolkit for evaluating multimodal models across image, video, and audio tasks, with 100+ benchmarks and 30+ model backends. lmms-eval is an evaluation toolkit for large multimodal models. It runs a model against a shared set of benchmarks covering image, video, and audio understanding, so teams can compare results on the same tasks with the same pipeline instead of stitching together scattered datasets and post-processing scripts.
- Is lmms-eval free to use? What license does it have?
- lmms-eval is open source under the Other license. Check the license terms before using it in commercial products.
- Is lmms-eval still maintained?
- Actively maintained. The last commit to lmms-eval was on Nov 6, 2026.
- What can I use lmms-eval for?
- Comparing a multimodal model against standard image or video benchmarks. Evaluating training checkpoints automatically during model training. Benchmarking API-hosted models such as GPT-4o alongside open-weight models. Adding an in-house evaluation dataset to a shared, reproducible pipeline.
- How do I install lmms-eval?
- Clone the repository: git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git Install the package in editable mode with uv: cd lmms-eval && uv pip install -e ".[all]" Run a quick evaluation (Qwen2.5-VL on MME, 8 samples): python -m lmms_eval --model qwen2_5_vl --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct --tasks mme --batch_size 1 --limit 8
- Where can I get lmms-eval?
- The source code and installation instructions are on GitHub: https://github.com/EvolvingLMMs-Lab/lmms-eval
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
A unified training engine for multimodal models, supporting VLM, diffusion and LLM training with FSDP2, Muon and Liger kernels.
83340PythonNo licenseUpdated 15 days ago
A Python framework for building cross-environment benchmarks that evaluate multimodal LLM agents across in-memory, Docker, VM, and physical-machine environments.
432+2 this week64PythonNo licenseUpdated 5 days ago
An open-source toolkit for running one-command evaluations of large vision-language models across many multimodal benchmarks.
4.5K+7 this week789PythonApache-2.0Well maintainedUpdated 21 hours ago
A visual, node-based environment for comparing prompts and models by querying multiple LLMs at once and scoring their responses.
AI & Machine Learning#prompt-engineering#llm-evaluation#large-language-models#visual-programming#multi-model-comparison
3K261TypeScriptMITUpdated 8 days ago
An open-source one-stop platform for evaluating large language models across 100+ datasets and many model providers and inference backends.
7.6K+10 this week903PythonApache-2.0Updated 20 hours ago