What is Time to first token?
Also called TTFT, first token latency.
Time to first token is the delay between sending a request to a language model and receiving the first piece of the streamed response. It captures queueing, prompt processing, and any preliminary work, but not the time spent generating the rest of the output. It is the main determinant of how responsive a streaming interface feels.
A streamed response arrives in pieces. Before the first piece appears, the request must be queued at the provider, and the entire prompt must be processed, a stage often called prefill. Time to first token therefore grows with prompt length and with how busy the provider is, while the remaining time depends on how many tokens the model generates afterward.
It matters because responsiveness is judged by when something starts happening, not by when it finishes. An interface that shows text within a second feels alive even if the full answer takes several seconds. This is why streaming, progressive disclosure of intermediate steps, and shorter system context are standard interface techniques rather than cosmetic details.
Two misreadings are common. The first is optimizing time to first token in isolation while total completion time gets worse, which trades a real gain for a perceived one. The second is treating it as a stable property when it fluctuates with provider load and prompt size. In multi-step agents there is also a gap between per-call and user-visible measures, because several hidden steps may run before any text appears.
Time to first token pairs with output token rate, sometimes called tokens per second, to describe streaming performance completely: one measures when the response starts, the other how fast it continues. Both are components of end to end latency. Caching a stable prompt prefix, trimming context, and starting to stream before all preparation finishes are the usual improvements.
Key points
- Measures responsiveness, not total completion time.
- Grows with prompt length and with provider queueing under load.
- Prefix caching and shorter prompts are the usual levers.
- In multi-step agents, hidden steps delay the first visible token.
In practice
A chat interface streams answers. For short questions the first word appears in about half a second, so the wait feels instant even though the full answer takes six seconds. After a change that prepends a long instruction block and a document, the first word takes four seconds. Total time barely moves, but users start reporting that the assistant feels slow and broken.