What is Training Data?
Also called training corpus.
Training data is the collection of text, code, images, or other content a model learns from during training. Its composition, quality, filtering, and licensing shape what a model knows, how it writes, and which biases it carries. Data selection is now regarded as comparable in importance to architecture and scale.
Large pre-training corpora are assembled from web crawls, books, code repositories, reference works, and licensed collections, then heavily processed. Typical steps include deduplication, quality classification, language identification, removal of boilerplate, and filtering for harmful content. A large share of raw crawled text is discarded, and the discard criteria materially change the resulting model.
Mixture proportions act as a design lever. Increasing the share of code has been observed to improve structured reasoning on non-code tasks; increasing the share of a language improves fluency in it; oversampling high-quality sources can outperform simply adding more low-quality volume. These proportions are among the most closely guarded details of a training recipe.
Training data creates the model's knowledge boundary. Anything absent from the corpus is unknown unless supplied at inference time, and anything present only in outdated form will be reproduced in that form. This is the underlying reason retrieval and tool use matter: they inject current or private information that no amount of training scale can supply.
Provenance carries legal and ethical weight. Copyright status, personal information, consent, and license compatibility are unresolved in many jurisdictions and subject to active litigation. Documentation practices such as data statements and model cards exist partly so that downstream users can assess these risks rather than inheriting them blind.
Key points
- Composition and filtering shape knowledge, style, and bias
- Most raw crawled text is discarded by quality filters
- Mixture proportions are a deliberate design lever
- Absent information cannot be recalled, only retrieved
- Provenance and licensing carry unresolved legal risk
In practice
Two models of similar size behave very differently on technical writing tasks. One was trained on a mixture weighted toward documentation, forums, and code; the other on a broader general web mixture. The first produces more accurate command syntax and adopts documentation conventions unprompted. Neither team changed architecture; the difference came entirely from what went into the corpus and what the filters removed.