What is Top-p Sampling?
Also called nucleus sampling, top-p.
Top-p sampling, also called nucleus sampling, limits the model's choice at each step to the smallest set of candidate tokens whose probabilities add up to a threshold p. One token is then drawn from that set alone. Unlike a fixed cutoff, the set size adapts: it stays small when the model is confident and grows when many continuations are plausible.
Suppose the threshold is set to 0.9. The model sorts candidate tokens by probability and adds them up until the running total reaches ninety percent, then discards everything below that line. If a single token already carries ninety percent of the probability, the eligible set contains just that one option. If a hundred tokens are needed to reach it, all hundred stay eligible.
The adaptive set is the entire point of the method. An older approach, top-k, always keeps a fixed number of candidates, which is too permissive where the next word is nearly determined and too restrictive where many words would fit. Nucleus sampling avoids both failure modes by responding to the model's own confidence at each individual position.
Values very close to one leave nearly every token eligible, including a long tail of implausible options that occasionally derail a response. Values below roughly 0.7 make text noticeably repetitive and bland, since the model keeps returning to its safest continuations. Most provider defaults sit between those extremes, and a great many applications never need to change them.
Top-p and temperature stack on each other, which is why changing both at once is discouraged. Temperature reshapes the whole distribution first, and top-p then truncates whatever remains, so the combined effect is difficult to predict or reproduce. Reasoning-focused models increasingly fix both settings internally and ignore values supplied by the caller.
Key points
- Keeps the smallest candidate set whose probabilities reach the threshold.
- The candidate set shrinks when confident and grows when uncertain.
- More adaptive than top-k, which keeps a fixed candidate count.
- Stacking it with temperature makes behavior hard to reason about.
In practice
The model is completing 'The capital of France is'. One token carries almost all of the probability, so at a threshold of 0.9 the eligible set holds a single option and the output is effectively fixed. Now complete 'My favorite season is'. Four options split the probability fairly evenly, the set holds all four, and repeated runs give different answers.