Repolode
Submit
cvs-health

cvs-health/uqlm

Python library that scores LLM responses for likely hallucinations using uncertainty quantification, with black-box, white-box, judge and ensemble scorers.

View on GitHubWebsite
Stars
1,217
Forks
140
Open issues
26
Last commit
25 days ago

What is it?

UQLM is a Python library for detecting hallucinations in LLM outputs. Each scorer returns a confidence score between 0 and 1, where higher values mean a response is less likely to contain errors. It helps developers flag unreliable answers without needing a ground-truth reference for every prompt.

The package groups its scorers by how they get their signal. Black-box scorers sample several responses to the same prompt and measure how consistent they are, so they work with any LLM. White-box scorers read token probabilities in a single generation and are cheaper but need API access to those probabilities. LLM-as-a-judge scorers ask one or more models to evaluate a response, and ensemble scorers combine several scorers via weighted averaging.

UQLM also includes a long-text scorer that scores individual claims in a long response and can remove low-confidence claims to refine the output, plus a CodeGenUQ interface for predicting whether generated code is functionally correct without executing it. Models are accessed through LangChain Chat Model integrations.

Key features

  • 01Black-box scorers measure consistency across multiple sampled responses.
  • 02White-box scorers compute confidence from token probabilities in a single generation.
  • 03LLM-as-a-judge scorers use one or more judge models, including a panel of judges.
  • 04Ensemble scorers combine multiple scorers, with off-the-shelf or tuned weights.
  • 05Long-text scorers score individual claims and can refine responses by removing low-confidence claims.
  • 06Works with any LangChain Chat Model.

Use cases

  • Flagging unreliable LLM answers in a production question-answering pipeline.
  • Filtering or refining long generated documents by removing low-confidence claims.
  • Estimating whether LLM-generated code is likely functionally correct before running it.
  • Comparing hallucination detection methods on your own prompts and labeled answers.

Quick start

  1. 1

    Install uqlm from PyPI

    pip install uqlm
  2. 2

    Create a LangChain chat model (e.g. ChatOpenAI) and wrap it with BlackBoxUQ to score responses

Things to know

  • Black-box and judge scorers make multiple LLM calls, which adds noticeable cost and latency.
  • White-box scorers need access to token probabilities, which many hosted APIs do not expose.

In collections

Testing frameworks & test automation tools

Topics

#hallucination-detection#uncertainty-quantification#llm-evaluation#confidence-estimation#llm-safety#python#ai-evaluation#ai-safety#confidence-score#hallucination

FAQ

What is uqlm?
Python library that scores LLM responses for likely hallucinations using uncertainty quantification, with black-box, white-box, judge and ensemble scorers. UQLM is a Python library for detecting hallucinations in LLM outputs. Each scorer returns a confidence score between 0 and 1, where higher values mean a response is less likely to contain errors. It helps developers flag unreliable answers without needing a ground-truth reference for every prompt.
Is uqlm free to use? What license does it have?
uqlm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is uqlm still maintained?
Actively maintained. The last commit to uqlm was on Oct 13, 2026.
What can I use uqlm for?
Flagging unreliable LLM answers in a production question-answering pipeline. Filtering or refining long generated documents by removing low-confidence claims. Estimating whether LLM-generated code is likely functionally correct before running it. Comparing hallucination detection methods on your own prompts and labeled answers.
How do I install uqlm?
Install uqlm from PyPI: pip install uqlm Create a LangChain chat model (e.g. ChatOpenAI) and wrap it with BlackBoxUQ to score responses
Where can I get uqlm?
The source code and installation instructions are on GitHub: https://github.com/cvs-health/uqlm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects