Azure-Samples/ai-rag-chat-evaluator

What is it?

Key features

• 01Scores RAG answers with GPT-based metrics such as groundedness, relevance, and coherence.
• 02Computes code-based metrics including latency, answer length, and citation checks.
• 03Generates ground truth question sets, including questions the documents cannot answer.
• 04Compares answers across runs in a terminal diff view, with filtering by changed metrics.
• 05Supports Azure OpenAI with keyless authentication or openai.com accounts.

Use cases

• Checking whether a prompt or search setting change improves answer quality before deploying.
• Measuring how often a RAG chat app admits it does not know when the sources lack an answer.
• Running quick subset evaluations during development with a limited number of questions.
• Comparing two experiment runs question by question to find regressions.

Quick start

1. 1

   Install Python 3.10 or higher

2. 2

   Create and activate a Python virtual environment

3. 3

   Install the project into the virtual environment

   python -m pip install -e .
4. 4

   Run an evaluation with the example config

   python -m evaltools evaluate --config=example_config.json

Things to know

• !Evaluation relies on a GPT-4 class model, so you need an Azure OpenAI or openai.com instance.
• !Running GPT-based evaluators incurs token costs for every question and evaluator used.
• !Built-in metrics use English-language prompts and are only suited to English answers.

In collections

Topics

FAQ

What is ai-rag-chat-evaluator?

Scripts and tools to evaluate RAG chat apps using the Azure AI Evaluate SDK and OpenAI-based graders. This project provides a command-line toolkit for measuring the answer quality of chat applications built on retrieval-augmented generation (RAG). Parameters such as the system prompt, search settings, and GPT model options all change how answers look, so each change can be checked against a set of ground truth question-and-answer pairs.

Is ai-rag-chat-evaluator free to use? What license does it have?

ai-rag-chat-evaluator is open source under the MIT license. Check the license terms before using it in commercial products.

Is ai-rag-chat-evaluator still maintained?

Actively maintained. The last commit to ai-rag-chat-evaluator was on Nov 3, 2026.

What can I use ai-rag-chat-evaluator for?

Checking whether a prompt or search setting change improves answer quality before deploying. Measuring how often a RAG chat app admits it does not know when the sources lack an answer. Running quick subset evaluations during development with a limited number of questions. Comparing two experiment runs question by question to find regressions.

How do I install ai-rag-chat-evaluator?

Install Python 3.10 or higher Create and activate a Python virtual environment Install the project into the virtual environment: python -m pip install -e . Run an evaluation with the example config: python -m evaltools evaluate --config=example_config.json

Where can I get ai-rag-chat-evaluator?

The source code and installation instructions are on GitHub: https://github.com/Azure-Samples/ai-rag-chat-evaluator