# What is Speculative Decoding?

Also called assisted generation, draft-and-verify decoding.

Speculative decoding is an inference optimization in which a small fast model drafts several tokens ahead and a larger target model verifies them in a single parallel pass. Accepted draft tokens are kept and the first rejected one is corrected by the target model. It speeds up generation without changing the output distribution of the target model.

Generation is normally memory-bandwidth bound rather than compute bound: producing one token requires reading the entire set of weights, and the arithmetic involved barely occupies the hardware. Verifying several candidate tokens at once costs almost the same as verifying one. Speculative decoding exploits that gap by giving the expensive model several tokens to check per weight read.

The correctness guarantee comes from a rejection sampling procedure. Each drafted token is accepted with a probability derived from the ratio of the target and draft distributions, and on rejection a corrected token is sampled from an adjusted distribution. Done properly, the sequence distribution is identical to what the target model would have produced alone, so this is a pure speed optimization rather than a quality trade.

Speedup depends on how often the draft is accepted. Predictable text such as boilerplate, code with strong conventions, or content quoted from the prompt yields long accepted runs. Novel or high-entropy text yields frequent rejections, and each rejection wastes the drafting work. Real-world gains typically land somewhere between modest and roughly threefold depending on workload.

Variants remove the separate draft model. Self-speculative approaches use a subset of the target model's own layers as the drafter. Lookup approaches propose continuations copied from the prompt or from an n-gram cache, which works remarkably well for editing and summarization tasks where output overlaps the input heavily.

## Key points

- Small model drafts tokens, large model verifies in parallel
- Exploits memory bandwidth limits in token generation
- Preserves the target model's exact output distribution
- Speedup depends on how predictable the text is
- Draft-free variants reuse layers or prompt n-grams

## In practice

A user asks the system to reformat a long block of configuration text. Most output tokens are copied straight from the input, so a lookup drafter proposes runs of them and the target model accepts nearly all in each verification pass. Throughput roughly doubles. When the same user then asks for an original explanation of what the configuration does, acceptance drops sharply and the speedup narrows, because little of the output is predictable from the input.

## Related terms

- [Inference](/en/glossary/inference)
- [Small Language Model](/en/glossary/small-language-model)
- [KV Cache](/en/glossary/kv-cache)
- [Greedy Decoding](/en/glossary/greedy-decoding)
- [Model Routing](/en/glossary/model-routing)

[Back to the AI Glossary](/en/glossary)