What is Perplexity?
Perplexity is a metric for how well a language model predicts a body of text, computed as the exponential of the average negative log probability the model assigns to each token. Lower values mean the model found the text less surprising. It is the standard intrinsic measure used during pre-training and for comparing models on the same evaluation data.
Intuitively, perplexity is the effective number of equally likely options the model was choosing among at each step. A perplexity of ten means the model was about as uncertain as if picking uniformly from ten candidates. A perfect model that assigned probability one to every observed token would score one; a model assigning uniform probability across the vocabulary would score the vocabulary size.
Its main use is tracking training progress and comparing architectures or data mixtures. Because it is computed directly from the model's own probabilities, it needs no labels, no human judgment, and no task design. Every checkpoint can be scored cheaply on a held-out set, which makes it the workhorse metric of pre-training research.
Comparisons across models are treacherous. Perplexity depends on the tokenizer, since the same text split into different numbers of tokens produces different per-token averages. It also depends on the evaluation text, its domain, and its length. Two published perplexity figures are only comparable if the tokenizer and the exact evaluation corpus match.
Low perplexity does not imply usefulness. A model can predict text extremely well and still fail at instruction following, refuse poorly, or reason badly, because none of those qualities is what the metric measures. Instruction tuning and preference training often raise perplexity on generic text while sharply improving practical helpfulness, so downstream evaluation remains necessary.
Key points
- Exponential of average negative log probability per token
- Lower means the model found the text less surprising
- Cheap to compute and needs no labels
- Not comparable across different tokenizers or corpora
- Low perplexity does not guarantee useful behavior
In practice
A team compares two data mixtures for a pre-training run by scoring both checkpoints on the same held-out set of technical documentation with the same tokenizer. The mixture with more code and manuals scores meaningfully lower perplexity on that set, confirming better fit to the target domain. They still run task benchmarks afterward, because the perplexity result says nothing about whether either checkpoint follows instructions.