# What is Retrieval Evaluation?

Also called retrieval benchmarking.

Retrieval evaluation measures how well a search system finds the documents needed to answer a set of test queries. It requires a labeled set pairing queries with known relevant passages, and it scores the retriever separately from the model that writes the final answer. Without it, pipeline changes are guesses.

The essential discipline is separating retrieval quality from generation quality. If an answer is wrong, either the right passage was never retrieved or it was retrieved and misused, and those two failures need different fixes. Measuring the retriever alone, on whether the relevant passage appeared in the top results, isolates the first case and makes the second diagnosable.

Building the labeled set is the real work. A few hundred queries drawn from actual user logs, each paired with the passages that genuinely answer them, is more valuable than thousands of synthetic pairs generated from the documents themselves. Model-generated pairs are useful for bulk coverage but tend to be phrased in the corpus vocabulary, which flatters retrieval and hides exactly the vocabulary mismatch problems real users hit.

Standard metrics come in complementary pairs. Hit rate and recall at a cutoff describe whether the needed passage was found at all. Mean reciprocal rank and normalized discounted cumulative gain describe how high it landed. Precision-oriented measures describe how much noise came with it. A change that improves recall while hurting rank position is common, so reporting one number alone is misleading.

Evaluation must be rerun on a fixed corpus snapshot whenever any component changes: chunk size, embedding model, index parameters, filters, fusion weights, or reranker. Recording the corpus version, the index configuration, and the metric definitions alongside each result is what makes comparisons across weeks meaningful rather than accidental.

## Key points

- Scores the retriever separately from answer generation
- Needs query and passage labels drawn from real usage
- Report both found-at-all and ranked-high metrics
- Synthetic query sets flatter retrieval and hide vocabulary gaps
- Results are only comparable against a fixed corpus snapshot

## In practice

A team assembles two hundred real support questions and marks which knowledge base passages answer each. Their baseline scores 0.72 hit rate at five. Adding a reranking step raises it to 0.86 and lifts mean reciprocal rank from 0.51 to 0.68. A later chunk-size change looks better on hit rate but drops mean reciprocal rank, revealing that answers were being pushed further down the list.

## Related terms

- [Recall and Precision](/en/glossary/recall-and-precision)
- [Mean Reciprocal Rank](/en/glossary/mean-reciprocal-rank)
- [Context Precision](/en/glossary/context-precision)
- [Faithfulness](/en/glossary/faithfulness)
- [Reranking](/en/glossary/reranking)

[Back to the AI Glossary](/en/glossary)