What is Prompt Cache Hit Rate?
Also called cache hit rate, prefix cache rate.
Prompt cache hit rate is the proportion of input tokens served from a provider's cached prefix rather than processed fresh. Many model providers cache the computed state of a repeated prompt prefix and meter those tokens differently from new ones. A higher hit rate typically reduces both latency and metered input cost.
Caching works on exact prefix matches. The cache holds the processed state of the leading portion of a prompt, so a request whose beginning is byte-identical to a previous one can reuse it and process only the divergent remainder. A single differing character early in the prompt invalidates everything after it, which is why the mechanic is so sensitive to prompt layout.
This makes ordering the dominant design factor. Stable content such as system instructions, tool definitions, and reference documents belongs at the front; volatile content such as the current timestamp, a session identifier, or a randomized greeting belongs at the end. A timestamp injected into the first line of a system prompt reliably reduces the hit rate to near zero regardless of how much stable text follows.
Caches also expire. Providers apply a time to live, so a prefix unused for long enough is evicted and the next request pays full processing. Traffic patterns therefore affect hit rate independently of prompt design: steady traffic over a shared prefix keeps it warm, while sparse or bursty traffic across many distinct prefixes repeatedly pays cold costs.
The metric is worth tracking per prompt template rather than only in aggregate. An overall rate averages together templates with very different behavior and hides the one whose recent edit broke caching. Provider responses commonly report cached and uncached input token counts per call, so the rate can be computed directly from the same usage records used for cost attribution.
Key points
- Fraction of input tokens served from a cached prompt prefix
- Caching depends on exact prefix match, so ordering matters
- Put stable instructions first and volatile values last
- Caches expire, so sparse traffic pays cold costs repeatedly
- Track per template, since aggregates hide broken cases
In practice
An agent's prompt begins with a system message, then tool definitions, then the conversation. A change adds the current time to the top of the system message, and within a day the cached share of input tokens across that template falls from 78 percent to under 5 percent. Moving the timestamp into a message appended just before the user's turn restores the previous rate.