# What is Pre-training?

Also called pretraining, base training.

Pre-training is the first and largest stage of building a model, in which it learns general patterns from a very large unlabeled corpus, typically by predicting the next token. It produces a base model with broad knowledge and language ability but no particular tendency to follow instructions. Later stages then adapt that base into a usable assistant.

The corpus is assembled from web text, books, code repositories, and licensed collections, then filtered heavily for duplicates, low quality, and unsafe material. No human labels are required, because the text supplies its own supervision: the next word is already the correct answer. That property is exactly what allows training at a scale no manual labeling effort could ever match.

Pre-training dominates the compute budget and runs for weeks or months across large clusters of accelerators. The knowledge cutoff comes from this stage, since the base model only ever sees documents gathered before the corpus was frozen. Anything later has to reach the model through the prompt, through a retrieval system, or through a subsequent training run on newer data.

A base model straight out of pre-training behaves unlike the assistants most people have used. Asked a question, it may continue the text as a list of similar questions, because continuation is precisely what it was trained to do. Instruction following, refusal behavior, and conversational formatting all come from later stages, not from this one, which regularly surprises newcomers.

Very few organizations perform pre-training, since it requires enormous capital, mature data pipelines, and specialized engineering. The overwhelming majority of practical work happens downstream through prompting, fine-tuning, retrieval, and tool use. This concentration is why so many independent products ultimately depend on a small number of base models and inherit their strengths and limitations together.

## Key points

- Largest and most expensive stage; consumes most of the compute budget.
- Uses unlabeled text, where the next token supplies its own supervision.
- Sets the knowledge cutoff date of the resulting model.
- Base models continue text and do not reliably follow instructions.

## In practice

Take a sentence from a technical manual with the final word hidden: 'Tighten the bolt to the specified ___'. The model guesses, is told the answer was 'torque', and its weights shift a fraction. Repeat that across trillions of words drawn from every subject, and the resulting base model has absorbed vocabulary, facts, and structure without anyone labeling a single example.

## Related terms

- [Fine-tuning](/en/glossary/fine-tuning)
- [Instruction Tuning](/en/glossary/instruction-tuning)
- [Parameters](/en/glossary/parameters)
- [Foundation Model](/en/glossary/foundation-model)
- [Large Language Model](/en/glossary/large-language-model)

[Back to the AI Glossary](/en/glossary)