Repolode
Submit
open-compass

open-compass/opencompass

An open-source one-stop platform for evaluating large language models across 100+ datasets and many model providers and inference backends.

View on GitHubWebsite
Stars
7,531
Forks
888
Open issues
376
Last commit
4 hours ago

What is it?

OpenCompass is a platform for benchmarking large language models. It supports models from providers such as OpenAI, Anthropic, Gemini, Qwen, GLM and DeepSeek, along with local open-source models, and covers more than 100 datasets spanning knowledge, reasoning, coding, science, language, long context and safety.

The project provides an end-to-end workflow: dataset and model configuration, inference, evaluation and result summarization. Inference can run through HuggingFace, LMDeploy or vLLM backends, and evaluation supports zero-shot, few-shot, chain-of-thought, rule-based and LLM-as-judge methods. Task partitioning, concurrent and distributed execution let it scale to different workloads.

OpenCompass also integrates VLMEvalKit so text-only and vision-language datasets can be run in one workflow. New models, datasets, evaluators and inference backends can be added through its extension points.

Key features

  • 01Supports API models such as OpenAI, Anthropic Claude and Google Gemini, and local open-source models.
  • 02Covers over 100 datasets across knowledge, reasoning, coding, science, long context and safety.
  • 03Runs inference through HuggingFace, LMDeploy or vLLM backends.
  • 04Offers rule-based, LLM-as-judge and math verification evaluators.
  • 05Supports task partitioning, concurrent and distributed execution.
  • 06Integrates VLMEvalKit for vision-language dataset evaluation.

Use cases

  • Comparing several open-source or API LLMs on a shared set of benchmarks.
  • Running reproducible evaluations of a model during development or fine-tuning.
  • Evaluating multimodal models on text and vision-language benchmarks in one workflow.

Quick start

  1. 1

    Create a conda environment with Python 3.12

    conda create --name opencompass python=3.12 -y
  2. 2

    Activate the environment

    conda activate opencompass
  3. 3

    Install OpenCompass via pip

    pip install -U opencompass
  4. 4

    Read the Quick Start guide in the documentation to run your first evaluation

Things to know

  • Some datasets (APPS, TACO, LiveCodeBench code generation) require Python 3.10 because of the pyext dependency.
  • Inference backends such as LMDeploy and vLLM must be installed separately because of conflicting dependencies.

Topics

#llm-evaluation#benchmark#large-language-model#inference#evaluation-framework#evaluation#llm#llm-eval

FAQ

What is opencompass?
An open-source one-stop platform for evaluating large language models across 100+ datasets and many model providers and inference backends. OpenCompass is a platform for benchmarking large language models. It supports models from providers such as OpenAI, Anthropic, Gemini, Qwen, GLM and DeepSeek, along with local open-source models, and covers more than 100 datasets spanning knowledge, reasoning, coding, science, language, long context and safety.
Is opencompass free to use? What license does it have?
opencompass is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is opencompass still maintained?
Actively maintained. The last commit to opencompass was on Oct 21, 2026.
What can I use opencompass for?
Comparing several open-source or API LLMs on a shared set of benchmarks. Running reproducible evaluations of a model during development or fine-tuning. Evaluating multimodal models on text and vision-language benchmarks in one workflow.
How do I install opencompass?
Create a conda environment with Python 3.12: conda create --name opencompass python=3.12 -y Activate the environment: conda activate opencompass Install OpenCompass via pip: pip install -U opencompass Read the Quick Start guide in the documentation to run your first evaluation
Where can I get opencompass?
The source code and installation instructions are on GitHub: https://github.com/open-compass/opencompass

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects