# What is Data Contamination?

Also called benchmark contamination, test set leakage.

Data contamination is the presence of evaluation material in a model's training data, which inflates measured performance because the model has effectively seen the answers. It is widespread with web-scraped corpora, since public benchmarks and their solutions are published online. Contamination makes reported scores unreliable as evidence of generalization.

The mechanism is straightforward. Benchmark questions, their answer keys, and discussions of them appear in repositories, papers, forums, and tutorials, all of which are routinely crawled. A model trained on that crawl may reproduce a memorized answer rather than solve the problem, and the resulting score reflects recall rather than the capability the benchmark was designed to measure.

Detection is imperfect. Exact string matching between training data and test items catches direct copies but misses paraphrases, translations, and reformatted variants. Behavioral probes look for suspicious signals such as a model completing a test item verbatim from its first few words, or scoring far better on a benchmark's public split than on a freshly written equivalent.

Mitigations are partial. Held-out benchmarks kept private, evaluation sets written after a model's training cutoff, dynamically generated problem instances, and canary strings embedded in test files all help. None fully solves the problem, because contamination can enter through fine-tuning data or through third-party corpora whose contents the model developer never fully inspects.

For anyone selecting a model, the operational conclusion is to weight public benchmark scores lightly and build a private evaluation set from real task data. A private set cannot have been trained on, measures the distribution actually cared about, and remains valid across model versions, which public leaderboards increasingly do not.

## Key points

- Evaluation items present in training data inflate scores
- Common because benchmarks are published on the open web
- Exact matching misses paraphrased and reformatted leakage
- Private, post-cutoff, or generated evaluations mitigate it
- Build a private evaluation set from real task data

## In practice

A model posts a strong score on a widely cited reasoning benchmark. A researcher rewrites the same problems with different names, numbers, and surface phrasing while preserving structure, and the score drops sharply. The original questions and their worked solutions appear in several public repositories, so the gap is consistent with memorization. The rewritten set, never published, becomes the team's internal reference instead.

## Related terms

- [Benchmark](/en/glossary/benchmark)
- [Training Data](/en/glossary/training-data)
- [Pre-training](/en/glossary/pre-training)
- [Synthetic Data](/en/glossary/synthetic-data)
- [Model Card](/en/glossary/model-card)

[Back to the AI Glossary](/en/glossary)