RAG retrieval evaluation.

Why evaluate retrieval separately from the answer?

Evaluate retrieval on its own because an end-to-end score cannot tell you which component failed. A RAG system is a pipeline: rewrite the query, retrieve candidates, optionally rerank, then generate from the top passages. When the answer is wrong, the cause is either that the right passage never arrived or that the model mishandled it, and the fixes are completely different.

The third row deserves attention. An answer that is right but unsupported by your documents is a liability in any setting where staff are told the system answers from approved sources. Our lesson on checking AI answers against sources explains why for non-technical users.

How do you build a retrieval evaluation dataset?

Build it from real questions with relevant passages labelled by people who know the documents. This is the slowest part of RAG evaluation and the part that pays back most.

1. Collect real questions. Pull 100 to 300 from search logs, support tickets, staff chat and interviews. Keep the original wording, typos included.
2. Pool candidates. For each question, gather the top results from several retrievers (keyword search such as BM25 plus one or two embedding setups) so labellers judge a varied set, not just what your current system returns.
3. Grade relevance. Mark each candidate 0 (not relevant), 1 (partially answers) or 2 (fully answers). Graded labels let you use nDCG, not just yes/no metrics.
4. Label spans, not chunks. Record relevance against the source document and character span. A chunk then counts as relevant if it overlaps a labelled span, so labels survive re-chunking.
5. Add unanswerable questions. Include questions whose answer is not in the corpus. The correct behaviour is a refusal or referral, and you need to measure it.
6. Double-label a sample. Have two people label the same 30 or so questions and resolve disagreements. Low agreement means the relevance guideline needs rewriting, not that the metric is wrong.

Label against stable spans from day one. Teams that label chunk IDs discover, the first time they change chunk size, that their whole dataset is invalid.

Which retrieval metrics should you use, and how are they defined?

Use recall@k to know whether the right passages arrive at all, and MRR or nDCG to know whether they arrive near the top. All are averaged across answerable questions; score unanswerable questions separately.

import math

def recall_at_k(ranked, relevant, k):
    if not relevant:
        return None                      # unanswerable: scored separately
    return len(set(ranked[:k]) & set(relevant)) / len(relevant)

def mrr(ranked, relevant, k=10):
    for rank, doc in enumerate(ranked[:k], start=1):
        if doc in relevant:
            return 1 / rank
    return 0.0

def ndcg_at_k(ranked, grades, k):
    """grades maps doc_id to 0, 1 or 2."""
    gain = lambda g, i: (2 ** g - 1) / math.log2(i + 1)
    dcg = sum(gain(grades.get(d, 0), i) for i, d in enumerate(ranked[:k], start=1))
    ideal = sorted(grades.values(), reverse=True)[:k]
    idcg = sum(gain(g, i) for i, g in enumerate(ideal, start=1))
    return dcg / idcg if idcg else None

Worked example: a question has two relevant passages, A (grade 2) and B (grade 1). The system returns X, A, Y, B, Z. Recall@3 is 0.5 and recall@5 is 1.0. MRR is 0.5, because the first relevant passage sits at rank 2. DCG@5 is 3/log2(3) + 1/log2(5), about 2.32; the ideal ranking (A then B) scores about 3.63, so nDCG@5 is about 0.64. Moving A to rank 1 would lift both MRR and nDCG without changing recall, which is exactly what a reranker is for.

Always include a plain keyword baseline. The BEIR benchmark, which compared ten retrieval systems across 18 datasets, found BM25 a strong baseline for out-of-domain search. Business documents full of product codes, policy numbers and names are often out of domain for general embedding models.

How do you measure the answer once retrieval is right?

Score faithfulness, correctness and refusals as three separate numbers. Ragas defines faithfulness as the number of claims in the response supported by the retrieved context divided by the total number of claims, on a 0 to 1 scale. That is a useful measure of grounding, but it says nothing about whether the source itself is current or correct.

• Answer correctness compares the answer with a reference answer written by a domain expert, using a rubric or a calibrated judge.
• Citation precision is the share of citations that actually support the sentence they are attached to. Systems that cite, such as Lucent, a Levy Street venture that grounds investment research in citations, live or die on this number.
• Refusal accuracy is the share of unanswerable questions that the system correctly declines, alongside the share of answerable ones it wrongly refuses.

Passage order affects faithfulness. Liu et al. found models use information at the start and end of long contexts better than information in the middle, so put the strongest passages first and avoid padding the prompt with marginal ones.

Which changes move which metric?

Each retrieval lever tends to move a particular metric, so measure the one you expect to change.

The choice of index and filtering strategy also changes recall. Our guide to choosing a vector database shows how post-filtering in approximate indexes can quietly return fewer results than you asked for.

Where does RAG retrieval fail in production?

Production failures are mostly about the corpus changing under the index, not about the model.

• Stale index. A policy is updated but its embeddings are not. Track the gap between source modified time and index time, and alert on it.
• Permission leakage. Retrieval returns passages the user should not see. Run evaluation queries as several test users and assert on what each can retrieve. The OWASP LLM Top 10 for 2026 lists vector and embedding weaknesses as LLM09:2026.
• Near-duplicates crowding the top k. Five versions of the same procedure fill every slot. Deduplicate at ingestion and keep only the current version searchable.
• Bad extraction. Tables and scanned PDFs extract as noise, so numeric questions score zero recall. Check extraction quality before tuning embeddings; see AI data preparation.
• Mixed embedding versions. Re-embedding part of a corpus with a new model leaves two incompatible vector spaces in one index. Re-index everything, then switch.

How do you keep RAG evaluation running?

Run retrieval evaluation in CI on every change to chunking, embeddings, filters or index settings, because it is cheap and fast. Run answer-level evaluation nightly or before releases. In production, log retrieved passage IDs with every answer and send a weekly sample to labellers, then add the misses to the dataset.

This is the core loop in our RAG development work. For the decision of whether retrieval is the right approach at all, see RAG versus fine-tuning, and for internal search across sensitive documents, building private AI search. If your team is new to the pattern, start with our lesson on understanding retrieval augmented generation.

• Questions come from real users, with original wording kept
• Candidates are pooled from several retrievers before labelling
• Relevance is graded 0, 1, 2 and labelled against stable source spans
• Unanswerable questions are included and scored separately
• Recall@k is reported at the k actually sent to the model
• MRR or nDCG is tracked for ranking quality, with a keyword baseline
• Faithfulness, correctness, citation precision and refusal accuracy are separate scores
• Retrieval evaluation runs in CI on every index or chunking change
• Index freshness and per-user access are tested, not assumed
• Production misses are labelled and added to the dataset every month

Written by the Levy Street team in Wellington. Last updated 19 October 2026.