Repolode
Submit
confident-ai

confident-ai/deepeval

Open-source Python framework for unit-testing LLM apps, with metrics for RAG, agents, chatbots, voice agents and more, similar in spirit to Pytest.

View on GitHubWebsite
Stars
18,982
Forks
2,102
Open issues
751
Last commit
yesterday

What is it?

DeepEval is an open-source LLM evaluation framework for testing LLM applications. It works like Pytest but is designed for LLM outputs: you write test cases with an input, the application's actual output, and optional expected output or retrieval context, then score them against metrics. Many metrics, such as G-Eval, use an LLM as the judge, and the framework works with the LLM of your choice.

The library covers end-to-end evaluation of an app treated as a black box, as well as component-level and trajectory-based evaluation of AI agents, tool calls, retrieval steps and sub-agent handoffs. Ready-made metrics include answer relevancy, faithfulness, contextual precision and recall for RAG, task completion and tool correctness for agents, knowledge retention for multi-turn chatbots, and hallucination, bias and toxicity checks. Custom metrics can be built and plugged into the same workflow.

Tests run from the command line with `deepeval test run` and fit into CI/CD pipelines. Tracing integrations cover OpenAI, Anthropic, LangChain, LangGraph, LlamaIndex, CrewAI, Pydantic AI and others. Results can optionally be synced to the Confident AI platform for shareable reports.

Key features

  • 01Pytest-style unit tests for LLM outputs, run with `deepeval test run`.
  • 02Built-in metrics for RAG, AI agents, multi-turn chatbots, voice agents, MCP and multimodal outputs.
  • 03G-Eval and DAG metrics that use an LLM judge with custom criteria.
  • 04End-to-end, component-level and trajectory-based evaluation with tracing.
  • 05Integrations with OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI and other frameworks.
  • 06Synthetic dataset generation and prompt optimization based on eval results.

Use cases

  • Adding regression tests for a RAG pipeline to a CI/CD workflow.
  • Checking whether an AI agent completes its tasks and calls the right tools.
  • Evaluating a multi-turn customer support chatbot for role adherence and knowledge retention.
  • Comparing models or prompts before switching LLM providers.

Quick start

  1. 1

    Open the README's quickstart in Google Colab (no local install needed)

Things to know

  • Its LLM-as-a-judge metrics rely on an LLM of your choice, which may require an API key and incur usage costs.

Topics

#llm-evaluation#testing#rag#ai-agents#python#pytest#llm-as-a-judge#evaluation-framework#evaluation-metrics#llm-evaluation-framework

FAQ

What is deepeval?
Open-source Python framework for unit-testing LLM apps, with metrics for RAG, agents, chatbots, voice agents and more, similar in spirit to Pytest. DeepEval is an open-source LLM evaluation framework for testing LLM applications. It works like Pytest but is designed for LLM outputs: you write test cases with an input, the application's actual output, and optional expected output or retrieval context, then score them against metrics. Many metrics, such as G-Eval, use an LLM as the judge, and the framework works with the LLM of your choice.
Is deepeval free to use? What license does it have?
deepeval is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is deepeval still maintained?
Actively maintained. The last commit to deepeval was on Oct 19, 2026.
What can I use deepeval for?
Adding regression tests for a RAG pipeline to a CI/CD workflow. Checking whether an AI agent completes its tasks and calls the right tools. Evaluating a multi-turn customer support chatbot for role adherence and knowledge retention. Comparing models or prompts before switching LLM providers.
How do I install deepeval?
Open the README's quickstart in Google Colab (no local install needed)
Where can I get deepeval?
The source code and installation instructions are on GitHub: https://github.com/confident-ai/deepeval

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects