Evaluate retrieval before generation: measuring a RAG system

When a retrieval-augmented generation (RAG) assistant gets something wrong, the usual reaction is to swap the model or rewrite the prompt. In practice, many of those errors happen one step earlier: the passage that held the answer never reached the model. An excellent model with the wrong context produces a wrong answer, just better written.

That's why the first thing we measure in a RAG system isn't the final answer. It's retrieval.

Split the system into two questions

A RAG pipeline answers two different questions, and each needs its own metric:

1. Did retrieval find the right evidence? That depends on chunking, embeddings, lexical search, permission filters and reranking.
2. Did generation use that evidence correctly? That depends on the model, the prompt and the citation rules.

If you only measure the final answer, you can't tell which one failed. A quality drop after changing the chunking strategy looks exactly like a drop caused by a worse prompt.

Build the evaluation set before the system

An evaluation set is a list of real questions, each annotated with the document passages that contain the answer. It doesn't need to be huge to be useful: a few dozen well-chosen questions already expose most regressions, and the set grows over time.

Three details matter:

• Questions from real users, not questions written by someone who knows the document by heart. Users use synonyms, abbreviations and vague phrasing.
• Unanswerable questions. Include cases where the documents don't contain the information. The correct behavior is to refuse, and that needs measuring too.
• Stable identifiers. Each passage needs an ID that doesn't change when you reprocess documents, or the evaluation set breaks with every pipeline change.

{"id": "q-017", "question": "What is the penalty for early termination?", "relevant": ["contract-0442/§14.2"], "answerable": true}

Retrieval metrics

With the set in hand, two simple metrics cover most needs.

Recall@k measures the fraction of questions where at least one relevant passage appeared in the top k results. If recall@5 is 0.70, then for 30% of questions the model never saw the evidence and, at best, will refuse.

MRR (mean reciprocal rank) measures how high the first relevant passage ranks. It matters because models weigh what comes first in the context more heavily, and because you may want to reduce k to save tokens.

def recall_at_k(results: dict[str, list[str]], gold: dict[str, set[str]], k: int) -> float:
    hits = sum(1 for q, ranked in results.items() if gold[q] & set(ranked[:k]))
    return hits / len(results)
 
 
def mean_reciprocal_rank(results: dict[str, list[str]], gold: dict[str, set[str]]) -> float:
    total = 0.0
    for q, ranked in results.items():
        for position, chunk_id in enumerate(ranked, start=1):
            if chunk_id in gold[q]:
                total += 1.0 / position
                break
    return total / len(results)

Compute these only over answerable questions. Unanswerable ones belong to the generation evaluation.

Generation metrics

For generation, three measures cover the main risks:

• Faithfulness: is every statement in the answer supported by the passages provided? It can be assessed by human review on samples and, at scale, by a judge model with strict instructions.
• Citation validity: does every citation point to a passage that was actually sent to the model? This is checkable in code, with no model involved, and should block the answer when it fails.
• Correct refusal rate: on unanswerable questions, did the system refuse? On answerable ones, did it refuse when it shouldn't have?

LLM-based judges are useful but biased: they tend to favor longer answers and can agree with plausible mistakes. Calibrate the judge against a human-reviewed sample before trusting its numbers.

Put evaluation in CI

Evaluation only protects the system if it always runs. We treat the evaluation set as a regression test: every change to chunking, embeddings, model or prompt runs the suite and compares against the baseline.

Two details prevent frustration:

1. Thresholds with headroom. Small variations are normal, especially in generation. Fail the build on meaningful drops, not on noise.
2. Per-question reports. An average hides where the problem is. Keep track of which questions flipped from pass to fail, because that's where the diagnosis lives.

When recall is low

A few causes come up often:

• Exact terms. Clause numbers, product codes and proper names are poorly represented by embeddings. Hybrid search, combining vectors with lexical search, usually helps.
• Chunks that cut content apart. Fixed token windows separate a rule from its exception. Segmenting along the document's structure helps.
• Permission filters applied late. If you filter after retrieving the top k, you may end up with nothing. Apply the filter inside the query itself.

Summary

Measuring retrieval separately turns RAG improvement from trial and error into engineering. With a versioned evaluation set, recall and faithfulness metrics and continuous execution, every change becomes an informed decision, and swapping a model stops being a gamble.