Repolode
Submit
braintrustdata

braintrustdata/autoevals

A Python and TypeScript library for scoring AI model outputs with LLM-as-a-judge, heuristic, and statistical evaluators.

View on GitHub
Stars
1,073
Forks
101
Open issues
59
Last commit
8 days ago

What is it?

Autoevals is a library for automatically evaluating the outputs of AI models. It gives you a uniform interface: pass in an input, an output, and an expected value, and get back a score between 0 and 1 from one of many built-in evaluators.

The library bundles three families of methods: model-graded evaluation where an LLM acts as a judge (for tasks such as factuality, safety, summarization, SQL and translation), heuristic checks like Levenshtein distance, exact match and JSON diff, and statistical measures such as BLEU. It also includes RAG-specific scorers for faithfulness, context recall and answer relevancy, along with embedding similarity.

Custom evaluations are supported through custom prompt templates with configurable choice scores, or through plain functions that return a Score object. By default it authenticates with OPENAI_API_KEY, and it can be pointed at any OpenAI-compatible endpoint, including the Braintrust AI Gateway. Results can optionally be logged to Braintrust.

Key features

  • 01Model-graded evaluators such as Factuality, Closed QA, Moderation and SQL
  • 02Heuristic scorers including Levenshtein distance, exact match and JSON diff
  • 03RAG scorers for faithfulness, context recall and answer relevancy
  • 04Custom prompt templates and custom Python or TypeScript scoring functions
  • 05Works with any OpenAI-compatible API, with sync and async calls

Use cases

  • Checking whether an LLM answer is factually consistent with a reference answer
  • Scoring RAG pipeline retrieval and answer quality during development
  • Comparing prompt or model versions with consistent metrics in a test suite
  • Adding custom LLM-judged or rule-based checks to an existing evaluation workflow

Quick start

  1. 1

    Install the Python package with pip

    pip install autoevals
  2. 2

    Set your OPENAI_API_KEY environment variable (default authentication)

  3. 3

    Run a Factuality evaluation from Python

Things to know

  • LLM-based evaluators need an API key and make paid model calls by default, though the Braintrust Gateway is free during beta.

In collections

Testing frameworks & test automation tools

Topics

#llm-evaluation#evals#llm-as-a-judge#rag#testing#openai#python#typescript

FAQ

What is autoevals?
A Python and TypeScript library for scoring AI model outputs with LLM-as-a-judge, heuristic, and statistical evaluators. Autoevals is a library for automatically evaluating the outputs of AI models. It gives you a uniform interface: pass in an input, an output, and an expected value, and get back a score between 0 and 1 from one of many built-in evaluators.
Is autoevals free to use? What license does it have?
autoevals is open source under the MIT license. Check the license terms before using it in commercial products.
Is autoevals still maintained?
Actively maintained. The last commit to autoevals was on Oct 30, 2026.
What can I use autoevals for?
Checking whether an LLM answer is factually consistent with a reference answer Scoring RAG pipeline retrieval and answer quality during development Comparing prompt or model versions with consistent metrics in a test suite Adding custom LLM-judged or rule-based checks to an existing evaluation workflow
How do I install autoevals?
Install the Python package with pip: pip install autoevals Set your OPENAI_API_KEY environment variable (default authentication) Run a Factuality evaluation from Python
Where can I get autoevals?
The source code and installation instructions are on GitHub: https://github.com/braintrustdata/autoevals

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects