# What is Corpus?

Also called document collection.

A corpus is the complete body of documents a retrieval system searches over. It defines the boundary of what can be found: nothing outside it is retrievable, however capable the model. Corpus composition, freshness, coverage, and duplication shape retrieval quality more than most tuning decisions applied downstream.

Composition is the first quality lever. A corpus assembled from whatever was easy to export usually contains outdated drafts, near-duplicate exports of the same policy, machine-generated logs nobody reads, and gaps in the areas users ask about most. Retrieval faithfully reflects that mixture, so an assistant that keeps citing a superseded document is usually revealing a corpus problem rather than a ranking problem.

Duplication is the most common and most damaging defect. When the same content appears in several files, near-identical chunks fill the top results and crowd out complementary sources, which narrows the evidence an answer is built from. Deduplication at ingestion, by content hash for exact copies and by similarity threshold for near copies, is usually worth more than any reranking change.

Freshness and supersession need an explicit policy. Someone has to decide what happens when a document is revised: whether the old version is deleted, retained with an expiry date, or kept but demoted. Without that decision the corpus accumulates contradictory statements, and retrieval will occasionally choose the wrong one, with no signal to the reader that a newer version exists.

Corpus statistics also underpin ranking itself. Lexical scoring depends on how rare a term is across the collection, so adding a large batch of documents shifts those weights. Evaluation sets drawn from an old snapshot stop being representative as the corpus grows, which is why retrieval metrics should record the corpus version they were measured against.

## Key points

- Defines the outer boundary of what can be retrieved
- Duplicates crowd top results and narrow the evidence base
- Supersession policy prevents contradictory answers
- Lexical scoring depends on collection-wide term statistics
- Evaluation results are tied to a specific corpus snapshot

## In practice

An assistant keeps quoting an eighteen-month-old pricing page. Inspection shows the corpus holds six exports of that page across two shared drives, all still marked current, versus one copy of the replacement. Retrieval was working correctly; the corpus was the defect. Deduplicating by content hash and deleting the superseded exports fixed the answers without any change to the retrieval code.

## Related terms

- [Knowledge Base](/en/glossary/knowledge-base)
- [Document Ingestion](/en/glossary/document-ingestion)
- [Indexing](/en/glossary/indexing)
- [Retrieval Evaluation](/en/glossary/retrieval-evaluation)
- [BM25](/en/glossary/bm25)

[Back to the AI Glossary](/en/glossary)