What is Decoder-Only Model?
Also called causal language model, autoregressive model.
A decoder-only model is a transformer architecture built from a single stack of causally masked layers that predicts each token from the tokens before it. It has no separate encoder, so input and output occupy the same sequence and the same processing path. Most contemporary text generation systems use this architecture.
The architecture treats every task as continuation. A prompt is placed at the start of the sequence, and the model repeatedly predicts the next token, appending each prediction before predicting again. Because the causal mask blocks any position from seeing later positions, the same forward pass that scores training data can be reused for generation without structural changes.
This uniformity is the architecture's main practical advantage. One objective, next-token prediction, covers summarization, translation, question answering, and code generation, so a single pre-training run produces broad capability. It also simplifies serving, since prompt processing and generation share one code path and one set of weights.
The trade-off is that the input is never processed bidirectionally. Tokens early in the prompt cannot be influenced by tokens that appear later, which is theoretically weaker than an encoder that reads the whole input at once. In practice, scale and instruction tuning have largely closed the resulting quality gap for generative tasks, though bidirectional encoders remain competitive for pure representation work.
Serving efficiency depends heavily on caching. Because each new token only needs to attend to previous positions, the intermediate key and value tensors for prior tokens can be stored and reused instead of recomputed. This makes generation cost roughly linear per token after the initial prompt pass, at the price of substantial memory for the cache.
Key points
- One causally masked stack handles both prompt and output
- Every task is framed as next-token continuation
- No bidirectional reading of the input sequence
- Reuses cached keys and values to generate efficiently
- The dominant architecture for text generation systems
In practice
Asked to summarize a report, a decoder-only system receives the report and the instruction as one token sequence, then emits the summary token by token onto the end of that same sequence. There is no separate encoding stage; the report is simply earlier context. This is why the prompt and the generated answer both consume the same context budget and are billed against the same token count.