Definition
A KV cache stores the attention keys and values already computed for earlier tokens at each transformer layer. When an autoregressive model generates the next token, it computes new attention state for the latest position and reuses the stored keys and values for the retained prefix instead of recomputing that prefix at every step.
The cache cuts repeated computation during token-by-token decoding, which usually improves latency. The tradeoff is memory. Cache use grows with the retained sequence, batch, number of layers, and attention configuration. In a serving system, KV-cache capacity can therefore limit how many long requests run together even when model weights fit comfortably on the accelerator.
Where the name comes from
"K" and "V" refer to the key and value tensors in attention. The 2017 Transformer paper formalized attention using query, key, and value projections. The cache is an inference optimization built around that structure, not a separate kind of model memory that understands or summarizes the conversation.
Inside one run and across requests
Model runtimes commonly maintain a KV cache during one generation. Providers may also reuse work for identical prompt prefixes across requests. That second feature is often called prompt caching or prefix caching. It may be implemented with KV state, but its lifetime, billing, routing, and invalidation rules are provider or runtime policies.
Yichao "Peak" Ji's Manus account explains the operational consequence for agents: a stable prefix and deterministic, append-only serialization can preserve cache reuse, while an early timestamp or reordered object can invalidate the reusable suffix. That is a production design pattern, not a guarantee shared by every API.
Distinguish it from nearby terms
- Working memory is information an agent or application keeps for the task. A KV cache is a numerical runtime artifact.
- Prompt caching reuses prior processing across requests under an implementation-specific policy.
- Context compression reduces or transforms the tokens supplied to the model. KV caching keeps computations for retained tokens without shortening the context.
Check your understanding
An agent changes one token near the start of a 100,000-token prefix. Explain why the visible content is almost identical while reuse after the changed token may disappear.