What is KV Cache?
Also called key-value cache, attention cache.
A KV cache is the stored set of key and value tensors computed for tokens already processed during generation, kept in memory so they do not have to be recomputed for each new token. It converts generation from a repeated full-sequence computation into an incremental one. Its memory footprint grows with sequence length and is a primary limit on serving capacity.
In a causally masked model, each new token attends to all previous tokens but the previous tokens never change. Their key and value projections are therefore fixed once computed. Caching them means each generation step only computes projections for the single new token and attends against the stored history, rather than reprocessing the whole sequence.
The efficiency gain is large. Without a cache, generating a sequence of length n would cost roughly the square of n in redundant work. With one, the prompt is processed once in a parallel prefill phase and each subsequent token costs a single incremental step. This is why the first token of a response typically takes noticeably longer than later ones.
The cost is memory. Cache size scales with sequence length, batch size, layer count, and attention head dimension, and for long contexts it can exceed the memory used by the weights themselves. This is what makes long conversations expensive to serve and why concurrent request limits often bind on cache memory rather than raw compute.
Several techniques target that pressure. Grouped and multi-query attention share key and value projections across heads, cutting cache size substantially. Cache quantization stores entries at lower precision. Paged allocation manages cache memory in fixed blocks to reduce fragmentation, and eviction or compression strategies drop or summarize older entries at some cost to fidelity.
Key points
- Stores key and value tensors for already-processed tokens
- Turns generation from quadratic into incremental work
- Explains why the first token is slower than the rest
- Memory grows with context length, batch size, and layers
- Grouped-query attention and paging reduce cache pressure
In practice
A user pastes a long document and asks three follow-up questions. The first request pays a full prefill over the document, building a large cache; each generated token afterward only extends it. If the serving system can retain that cache between turns, the follow-up questions skip re-reading the document entirely. If it cannot, every turn pays the prefill again, which is why cache retention is such a visible factor in long-conversation latency.