Repolode
Submit
modelscope

modelscope/evalscope

A customizable framework for evaluating LLMs, VLMs and AIGC models on benchmarks, plus inference performance stress testing and result dashboards.

View on GitHubWebsite
Stars
3,571
Forks
531
Open issues
10
Last commit
5 hours ago

What is it?

EvalScope is a framework for evaluating large language models and other generative models. It can run built-in benchmarks such as MMLU, C-Eval and GSM8K against an OpenAI API-compatible endpoint or a local model downloaded from ModelScope, and it reports scores in the terminal. Beyond accuracy, it includes a stress testing tool that measures throughput, time to first token (TTFT) and time per output token (TPOT) for model services.

The project also covers multimodal models, embedding and reranker models, and agent benchmarks run in a multi-turn loop with optional Docker sandboxes. Results can be explored in a web dashboard for side-by-side model comparison, and an Arena mode ranks models through pairwise battles. Third-party backends such as OpenCompass, VLMEvalKit and RAGEval can be launched through the same entry point.

Evaluation is configured via the command line, a Python API, or YAML/JSON files, and developers can add custom datasets, models and metrics.

Key features

  • 01Runs built-in benchmarks such as MMLU, C-Eval and GSM8K from a single command.
  • 02Evaluates OpenAI API-compatible endpoints and local models downloaded from ModelScope.
  • 03Stress tests model services and reports TTFT, TPOT and throughput.
  • 04Provides a web dashboard for comparing models and inspecting predictions.
  • 05Supports Arena-style pairwise battles with win rates and rankings.
  • 06Lets developers add custom datasets, models and metrics.

Use cases

  • Comparing several candidate LLMs on the same benchmark before choosing one.
  • Measuring latency and throughput of a model serving deployment under load.
  • Checking whether a third-party API deployment matches an official model's behavior.
  • Evaluating a fine-tuned model on a private dataset defined in a custom format.

Quick start

  1. 1

    Install the EvalScope package from PyPI

    pip install evalscope
  2. 2

    Run an evaluation against an OpenAI-compatible model API (set your endpoint and key first)

    evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5

Things to know

  • The quick-start example needs an OpenAI-compatible model API endpoint and API key.
  • Agent evaluation with sandboxed benchmarks such as SWE-bench requires Docker.
  • The project requires Python 3.10 or newer.

Topics

#llm-evaluation#benchmark#performance-testing#vlm#rag#python#model-evaluation#evaluation#llm#performance

FAQ

What is evalscope?
A customizable framework for evaluating LLMs, VLMs and AIGC models on benchmarks, plus inference performance stress testing and result dashboards. EvalScope is a framework for evaluating large language models and other generative models. It can run built-in benchmarks such as MMLU, C-Eval and GSM8K against an OpenAI API-compatible endpoint or a local model downloaded from ModelScope, and it reports scores in the terminal. Beyond accuracy, it includes a stress testing tool that measures throughput, time to first token (TTFT) and time per output token (TPOT) for model services.
Is evalscope free to use? What license does it have?
evalscope is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is evalscope still maintained?
Actively maintained. The last commit to evalscope was on Oct 21, 2026.
What can I use evalscope for?
Comparing several candidate LLMs on the same benchmark before choosing one. Measuring latency and throughput of a model serving deployment under load. Checking whether a third-party API deployment matches an official model's behavior. Evaluating a fine-tuned model on a private dataset defined in a custom format.
How do I install evalscope?
Install the EvalScope package from PyPI: pip install evalscope Run an evaluation against an OpenAI-compatible model API (set your endpoint and key first): evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5
Where can I get evalscope?
The source code and installation instructions are on GitHub: https://github.com/modelscope/evalscope

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects