Definition
Attention is a learned mechanism that scores relationships among elements and uses those scores to combine information into a new representation. It lets the model vary which source positions contribute to each output position instead of compressing every source into one fixed representation.
In a common form, the model compares a query with a set of keys, normalizes the resulting scores into weights, and uses those weights to combine corresponding values. Other attention mechanisms use different scoring functions or restrict which positions can be considered.
Self-attention draws queries, keys, and values from the same sequence or representation. Cross-attention takes the query from one representation and the keys and values from another, such as a text decoder attending to image features.
Development of the mechanism
In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio proposed a learned soft alignment mechanism for neural machine translation. It allowed the decoder to search the source sentence for information relevant to each predicted word. The 2017 transformer paper by Ashish Vaswani and coauthors made attention the architecture's primary way to mix information across positions and specified the scaled query-key-value calculation now widely used.
This history does not establish one inventor of every form of attention. Neural attention grew through several alignment, memory, and sequence-modeling lines of work.
Distinguish it from nearby terms
- Query-key-value attention is one formulation of attention, not the definition of every attention mechanism.
- Human attention concerns perception and cognition. A model's attention weights do not show awareness or intent.
- An explanation tells why a result should be trusted. Attention weights alone do not necessarily provide a faithful explanation of a model's decision.
Operational significance
Attention determines which positions can exchange information and contributes heavily to inference cost. The attention pattern, context length, masking rules, position encoding, and implementation all affect behavior. Seeing a high weight on the right token also does not prove that the model used it correctly downstream.
Check your understanding
A model assigns weight to the sentence containing the correct fact but answers incorrectly. That is not a contradiction: attention controls information mixing, not guaranteed recall, reasoning, or instruction compliance.