Is lm-evaluation-harness free to use? What license does it have?

lm-evaluation-harness is open source under the MIT license. Check the license terms before using it in commercial products.

Is lm-evaluation-harness still maintained?

Updated within 6 months. The last commit to lm-evaluation-harness was on Sep 14, 2026.

What can I use lm-evaluation-harness for?

Benchmarking a newly trained or fine-tuned model against standard academic tasks. Reproducing published evaluation numbers with the same prompts used in papers. Comparing quantized, LoRA-adapted or self-hosted models against hosted APIs under one setup. Running regression evaluations in CI when model checkpoints change.

How do I install lm-evaluation-harness?

Clone the repository: git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness Enter the project directory and install the package in editable mode: cd lm-evaluation-harness && pip install -e . Install the HuggingFace model backend: pip install "lm_eval[hf]" Evaluate a HuggingFace model on hellaswag: lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 --batch_size 8

Where can I get lm-evaluation-harness?

The source code and installation instructions are on GitHub: https://github.com/EleutherAI/lm-evaluation-harness

EleutherAI/lm-evaluation-harness

A unified framework for running few-shot evaluations of language models across dozens of standard academic benchmarks.

View on GitHub Website

Stars: 14,275
Forks: 3,698
Open issues: 1,212
Last commit: last month

What is it?

lm-evaluation-harness is a framework for evaluating generative language models on a large set of standard benchmarks. It implements over 60 academic benchmarks with hundreds of subtasks and variants, and uses publicly available prompts so that results can be compared across papers.

The harness supports a range of model backends, including Hugging Face transformers (with PEFT adapters and quantization), vLLM, SGLang, GGUF models served by llama.cpp, ONNX Runtime, NVIDIA NeMo, Megatron-LM, and hosted APIs such as OpenAI, Anthropic and TextSynth. Multi-GPU setups are covered through accelerate data parallelism, model sharding, and tensor parallelism. Tasks and prompts are defined in YAML, and the CLI supports subcommands and config files.

It is also the evaluation backend behind Hugging Face's Open LLM Leaderboard. Custom tasks, prompts and metrics can be added, and since version 2025 plugins can register new backends, filters and metrics from external packages.

Key features

01Covers over 60 standard academic LLM benchmarks with hundreds of subtasks.
02Supports Hugging Face transformers, vLLM, SGLang, llama.cpp, ONNX Runtime and NeMo backends.
03Evaluates hosted models from OpenAI, Anthropic, TextSynth and other API providers.
04Offers multi-GPU evaluation via data parallelism, model sharding and tensor parallelism.
05Defines tasks and prompts in YAML and runs them from a CLI or Python API.
06Allows custom tasks, metrics and model backends through configuration or plugins.

Use cases

Benchmarking a newly trained or fine-tuned model against standard academic tasks.
Reproducing published evaluation numbers with the same prompts used in papers.
Comparing quantized, LoRA-adapted or self-hosted models against hosted APIs under one setup.
Running regression evaluations in CI when model checkpoints change.

Quick start

1
Clone the repository
git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
2
Enter the project directory and install the package in editable mode
cd lm-evaluation-harness && pip install -e .
3
Install the HuggingFace model backend
pip install "lm_eval[hf]"
4
Evaluate a HuggingFace model on hellaswag
lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 --batch_size 8

Things to know

Model backends such as HuggingFace or vLLM must be installed separately, and the example run assumes a CUDA GPU.

Topics

#llm-evaluation #benchmark #language-model #few-shot #transformers #vllm #evaluation-framework #transformer

FAQ

What is lm-evaluation-harness?: A unified framework for running few-shot evaluations of language models across dozens of standard academic benchmarks. lm-evaluation-harness is a framework for evaluating generative language models on a large set of standard benchmarks. It implements over 60 academic benchmarks with hundreds of subtasks and variants, and uses publicly available prompts so that results can be compared across papers.
Is lm-evaluation-harness free to use? What license does it have?: lm-evaluation-harness is open source under the MIT license. Check the license terms before using it in commercial products.
Is lm-evaluation-harness still maintained?: Updated within 6 months. The last commit to lm-evaluation-harness was on Sep 14, 2026.
What can I use lm-evaluation-harness for?: Benchmarking a newly trained or fine-tuned model against standard academic tasks. Reproducing published evaluation numbers with the same prompts used in papers. Comparing quantized, LoRA-adapted or self-hosted models against hosted APIs under one setup. Running regression evaluations in CI when model checkpoints change.
How do I install lm-evaluation-harness?: Clone the repository: git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness Enter the project directory and install the package in editable mode: cd lm-evaluation-harness && pip install -e . Install the HuggingFace model backend: pip install "lm_eval[hf]" Evaluate a HuggingFace model on hellaswag: lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 --batch_size 8
Where can I get lm-evaluation-harness?: The source code and installation instructions are on GitHub: https://github.com/EleutherAI/lm-evaluation-harness

Summary generated by AI from the project's README. GitHub stats refresh daily.