Repolode
Submit
yixuantt

yixuantt/MultiHop-RAG

A QA dataset of 2,556 multi-hop queries for evaluating retrieval-augmented generation across multiple documents with metadata.

View on GitHub
Stars
476
Forks
46
Open issues
12
Last commit
last month

What is it?

MultiHop-RAG is a benchmark dataset for testing how well retrieval-augmented generation (RAG) systems handle questions that require combining evidence from several documents. It contains 2,556 queries, and the supporting evidence for each query is spread across 2 to 4 documents.

The queries also reference document metadata, so a system has to use that information during retrieval and reasoning, which mirrors the mixed-source situations common in real RAG applications. The project accompanies a COLM 2024 paper and is distributed both as files in the repository and through a Hugging Face dataloader.

The repository includes sample scripts for retrieval with llama-index (with optional reranking, using BAAI/llm-embedder) and for question answering with Llama, plus evaluation scripts for retrieval and QA results. Part of the dataset construction pipeline is also included for reference.

Key features

  • 01Contains 2,556 queries requiring evidence from 2 to 4 documents
  • 02Queries incorporate document metadata to reflect real-world RAG scenarios
  • 03Includes sample retrieval scripts with optional reranking
  • 04Provides a sample QA script using Llama
  • 05Ships evaluation scripts for retrieval and QA results

Use cases

  • Benchmarking retrieval quality of embedding models on multi-document questions
  • Testing whether an LLM can reason across several retrieved sources
  • Comparing RAG pipelines that use metadata filtering or reranking

Quick start

  1. 1

    Install the llama-index dependency

    pip install llama-index==0.9.40
  2. 2

    Run the simple retrieval sample

    python simple_retrieval.py --retriever BAAI/llm-embedder
  3. 3

    Run the QA sample with llama

    python qa_llama.py

Things to know

  • This is a dataset project rather than a standalone software tool, and the construction code is untidy and only provided for reference.
  • The repository states no software license in its metadata; the README names the ODC-BY data license.

In collections

Testing frameworks & test automation tools

Topics

#rag#dataset#benchmark#retrieval#multi-hop-qa#llm-evaluation

FAQ

What is MultiHop-RAG?
A QA dataset of 2,556 multi-hop queries for evaluating retrieval-augmented generation across multiple documents with metadata. MultiHop-RAG is a benchmark dataset for testing how well retrieval-augmented generation (RAG) systems handle questions that require combining evidence from several documents. It contains 2,556 queries, and the supporting evidence for each query is spread across 2 to 4 documents.
Is MultiHop-RAG free to use? What license does it have?
MultiHop-RAG has no declared license. Its code is public, but without a license you have no explicit permission to reuse it; ask the author before relying on it.
Is MultiHop-RAG still maintained?
Updated within 6 months. The last commit to MultiHop-RAG was on Sep 16, 2026.
What can I use MultiHop-RAG for?
Benchmarking retrieval quality of embedding models on multi-document questions Testing whether an LLM can reason across several retrieved sources Comparing RAG pipelines that use metadata filtering or reranking
How do I install MultiHop-RAG?
Install the llama-index dependency: pip install llama-index==0.9.40 Run the simple retrieval sample: python simple_retrieval.py --retriever BAAI/llm-embedder Run the QA sample with llama: python qa_llama.py
Where can I get MultiHop-RAG?
The source code and installation instructions are on GitHub: https://github.com/yixuantt/MultiHop-RAG

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects