Repolode
Submit
langchain-ai

langchain-ai/openevals

Python and TypeScript library of prebuilt and customizable evaluators for testing LLM application outputs, including LLM-as-judge scoring.

View on GitHub
Stars
1,227
Forks
133
Open issues
25
Last commit
1 hour ago

What is it?

OpenEvals is a library of ready-made evaluators for LLM applications, similar in spirit to unit tests for traditional software. It is meant as a starting point: teams can use the included evaluators directly and then write more specific ones for their own application.

The core is the create_llm_as_judge function, which takes a prompt and a judge model and returns an evaluator that formats inputs and outputs into the prompt and parses the judge's score. The package includes prebuilt prompts for quality, safety, security, RAG, code, structured extraction, agent trajectories, and voice or image content. Judge models can be specified as provider:model strings through LangChain or passed as a chat model or OpenAI client instance.

OpenEvals also ships non-LLM evaluators such as exact match, Levenshtein distance, and embedding similarity, and it integrates with LangSmith's pytest and Vitest/Jest runners. Python and TypeScript are both supported. Agent-specific evaluators live in a separate project, agentevals.

Key features

  • 01Creates LLM-as-judge evaluators from a prompt and a judge model.
  • 02Includes prebuilt prompts for quality, safety, security, RAG, and code checks.
  • 03Supports custom scoring values, output schemas, and multimodal attachments.
  • 04Provides non-LLM evaluators such as exact match, Levenshtein distance, and embedding similarity.
  • 05Works with LangSmith pytest and Vitest/Jest runners in Python and TypeScript.

Use cases

  • Checking whether an LLM chatbot's answers are concise, correct, or free of hallucinations before release.
  • Scoring RAG pipelines for groundedness and retrieval relevance.
  • Screening model outputs for toxicity, PII leakage, or prompt injection.
  • Adding LLM output checks to a CI test suite.

Quick start

  1. 1

    Install openevals with pip

    pip install openevals
  2. 2

    Set your OpenAI API key as an environment variable

    export OPENAI_API_KEY="your_openai_api_key"
  3. 3

    Run the conciseness evaluator example from the README

Things to know

  • The quickstart and default LLM-as-judge evaluators require an OpenAI API key, which is a paid service.

Topics

#llm-evaluation#evals#llm#testing#llm-as-judge#rag#langchain

FAQ

What is openevals?
Python and TypeScript library of prebuilt and customizable evaluators for testing LLM application outputs, including LLM-as-judge scoring. OpenEvals is a library of ready-made evaluators for LLM applications, similar in spirit to unit tests for traditional software. It is meant as a starting point: teams can use the included evaluators directly and then write more specific ones for their own application.
Is openevals free to use? What license does it have?
openevals is open source under the MIT license. Check the license terms before using it in commercial products.
Is openevals still maintained?
Actively maintained. The last commit to openevals was on Oct 21, 2026.
What can I use openevals for?
Checking whether an LLM chatbot's answers are concise, correct, or free of hallucinations before release. Scoring RAG pipelines for groundedness and retrieval relevance. Screening model outputs for toxicity, PII leakage, or prompt injection. Adding LLM output checks to a CI test suite.
How do I install openevals?
Install openevals with pip: pip install openevals Set your OpenAI API key as an environment variable: export OPENAI_API_KEY="your_openai_api_key" Run the conciseness evaluator example from the README
Where can I get openevals?
The source code and installation instructions are on GitHub: https://github.com/langchain-ai/openevals

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects