Definition
Stored attention keys and values from earlier tokens that an autoregressive transformer reuses instead of recomputing them for every new token.
Distinguish it from nearby terms
A KV cache accelerates token-by-token inference inside a model. Prompt caching may reuse a provider's prior work across requests and can include more than the model's attention state.
Check your understanding
The cache lowers repeated computation and latency, but consumes memory that grows with the retained sequence and batch.