Definition
Query-key-value attention is an attention formulation that compares queries with keys, normalizes the scores into weights, and applies those weights to corresponding values. Each output is a weighted mixture of value vectors selected according to how well their keys match the query.
In the transformer's scaled dot-product attention, the model computes query-key dot products, divides them by the square root of the key dimension to control their scale, applies a softmax, and multiplies the result by the values. Masks may block future or otherwise disallowed positions before normalization.
The query, key, and value vectors come from learned projections. In self-attention, all three are projected from the same sequence representation. In cross-attention, the queries come from one representation while the keys and values come from another. Multi-head attention repeats the calculation with separate projections so different heads can learn different relationship patterns.
Origin of the formulation
Ashish Vaswani and coauthors defined scaled dot-product attention and multi-head attention in the 2017 transformer paper Attention Is All You Need. The paper did not invent every use of queries, keys, values, or learned attention, but it established the formulation that modern transformer systems usually mean by QKV attention.
Distinguish it from nearby terms
- Attention is the broader family of mechanisms. QKV names a particular calculation within that family.
- Queries, keys, and values are learned vectors, not database queries, identifiers, or stored records. The retrieval analogy explains their roles but not their implementation.
- A KV cache stores previously computed key and value vectors during autoregressive inference. It reuses QKV work; it is not the attention mechanism itself.
Operational significance
For a sequence of length n, full self-attention constructs relationships across many pairs of positions, which can make memory and compute grow quickly as context expands. Implementations may use sparse, local, grouped-query, multi-query, or optimized kernels that preserve only part of the standard calculation. Architecture names should therefore be checked against the actual attention pattern and cache behavior.
Check your understanding
In cross-attention for image captioning, a text position can supply the query while image features supply keys and values. The query scores which image features match, and the corresponding value vectors carry the information mixed into the text representation.