Repolode
Submit
vectara

vectara/hallucination-leaderboard

A public leaderboard ranking LLMs by how often they hallucinate when summarizing short documents, scored with Vectara's HHEM evaluation model.

View on GitHubWebsite
Stars
3,322
Forks
107
Open issues
20
Last commit
4 days ago

What is it?

This repository hosts a public leaderboard that measures how often large language models introduce hallucinations when summarizing a document. Each model is given a document from a private dataset of over 7,700 articles and asked to summarize it using only facts from the source. Vectara's Hallucination Evaluation Model (HHEM) then checks whether each summary is factually consistent with the original text, producing a hallucination rate and a factual consistency rate per model.

The leaderboard also reports an answer rate, showing how often a model declined to summarize, and the average summary length, so that trivially short outputs cannot score well. Results are updated periodically as models and APIs change. The dataset is kept private to reduce overfitting, and an interactive version of the leaderboard is hosted on Hugging Face.

The repository documents the prompt, temperature setting, and the specific model versions and providers used for each entry, including Anthropic, OpenAI, Google Vertex AI, Mistral, Cohere, and several open-weight models accessed through hosted inference services.

Key features

  • 01Ranks dozens of LLMs by hallucination rate on document summarization.
  • 02Uses Vectara's HHEM model to score factual consistency of each summary.
  • 03Reports answer rate and average summary length alongside accuracy.
  • 04Documents the prompt, temperature, and exact model versions used.
  • 05Keeps the evaluation dataset private to reduce overfitting.

Use cases

  • Comparing candidate LLMs for summarization or RAG pipelines by factual reliability.
  • Choosing a model when faithfulness to source text matters more than fluency.
  • Tracking how hallucination rates change across model releases over time.

Things to know

  • The evaluation model HHEM-2.3 is Vectara's commercial model, while only an open-source variant (HHEM-2.1-Open) is publicly available.
  • The evaluation dataset is not publicly available, so the rankings cannot be independently reproduced from the data.

Topics

#llm#hallucinations#evaluation#summarization#benchmark#generative-ai

FAQ

What is hallucination-leaderboard?
A public leaderboard ranking LLMs by how often they hallucinate when summarizing short documents, scored with Vectara's HHEM evaluation model. This repository hosts a public leaderboard that measures how often large language models introduce hallucinations when summarizing a document. Each model is given a document from a private dataset of over 7,700 articles and asked to summarize it using only facts from the source. Vectara's Hallucination Evaluation Model (HHEM) then checks whether each summary is factually consistent with the original text, producing a hallucination rate and a factual consistency rate per model.
Is hallucination-leaderboard free to use? What license does it have?
hallucination-leaderboard is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is hallucination-leaderboard still maintained?
Actively maintained. The last commit to hallucination-leaderboard was on Oct 17, 2026.
What can I use hallucination-leaderboard for?
Comparing candidate LLMs for summarization or RAG pipelines by factual reliability. Choosing a model when faithfulness to source text matters more than fluency. Tracking how hallucination rates change across model releases over time.
Where can I get hallucination-leaderboard?
The source code and installation instructions are on GitHub: https://github.com/vectara/hallucination-leaderboard

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects