# What is Prompt Caching?

Also called Context Caching, Prefix Caching.

Prompt caching stores the processed form of a repeated prompt prefix so that subsequent requests reusing that prefix skip part of the computation. Because system instructions and retrieved reference material often stay identical across many calls, caching them reduces latency and cost for the unchanged portion while the varying tail is processed normally.

Model providers process a prompt by converting tokens into internal state before generating output. When the beginning of a prompt is identical to that of a recent request, that internal state can be reused instead of recomputed. Providers expose this behavior either automatically or through explicit cache markers placed in the request.

The mechanism only works on an exact prefix match, so prompt construction has to put stable content first and variable content last. A timestamp, a session identifier, or a reordered list of retrieved passages near the top of the prompt invalidates the cache for everything after it, which is a common and costly mistake.

Economics vary by provider, but the general shape is that writing to the cache costs slightly more than a normal request while reading from it costs substantially less, and entries expire after a short idle period. Caching therefore pays off for high frequency workloads sharing a large stable prefix and can cost more for occasional calls.

Caching changes cost and latency, not behavior, since the model produces the same output it would have produced without it. Care is still needed around isolation, because a cached prefix should never span tenants or users when it contains their data, and around invalidation when instructions or reference material are updated.

## Key points

- Reuses processed state for an identical prompt prefix.
- Stable content first, variable content last.
- Cache writes cost more, cache reads cost much less.
- Entries expire after a short idle window.
- Never share a cached prefix across tenants.

## In practice

An assistant sends a four thousand token system prompt plus a stable set of policy excerpts on every request, followed by the user's question. Marking that prefix as cacheable means the first call pays to populate the cache and the next several hundred calls reuse it, cutting time to first token noticeably while the answers stay identical.

## Related terms

- [Context Injection](/en/glossary/context-injection)
- [Retrieval Augmented Generation](/en/glossary/retrieval-augmented-generation)
- [Working Memory](/en/glossary/working-memory)
- [Agent Memory](/en/glossary/agent-memory)
- [Retrieval Pipeline](/en/glossary/retrieval-pipeline)

[Back to the AI Glossary](/en/glossary)