Definition
A transformer is a neural-network architecture that combines self-attention, position information, residual connections, normalization, and feed-forward transformations. Self-attention lets each position build a representation from other positions in the input according to learned query, key, and value projections.
Unlike recurrent networks, a transformer does not need to process every input position in sequence during training. That allows substantial parallel computation across the input. Autoregressive generation is still sequential at the output level because the model produces later tokens after earlier ones, though cached internal states can avoid recomputing the full prefix.
Transformers appear in several common forms. Encoder-only models build representations from an input. Decoder-only models generate sequences from left to right. Encoder-decoder models transform one sequence into another, as in the original machine-translation design. Vision, audio, protein, and multimodal systems can apply transformer blocks to units other than text tokens.
Where the name came from
Ashish Vaswani and seven coauthors introduced the Transformer architecture and name in their 2017 paper "Attention Is All You Need." Their model replaced sequence-aligned recurrence and convolution with attention in an encoder-decoder system. Attention itself was older. Bahdanau, Cho, and Bengio had already shown a learned attention mechanism for neural machine translation, and the Transformer paper cites earlier self-attention work. The 2017 contribution was the specific architecture built around that mechanism.
Distinguish it from nearby terms
- A transformer is an architecture family. A large language model describes a model's task and scale; many current LLMs use transformers, but the terms are not synonyms.
- Attention is a component used inside transformers and some non-transformer architectures.
- A decoder-only model is one transformer arrangement, not the definition of a transformer.
- A recurrent neural network carries state through sequence positions. A transformer relates positions primarily through attention and explicit position information.
Operational significance
Self-attention cost and memory can grow quickly with sequence length in the standard formulation. Implementations use restricted attention, sparse patterns, state caching, or other techniques when long inputs make full attention expensive. Architecture labels do not reveal context quality, training data, latency, or factual reliability, so those properties still need direct measurement.
Check your understanding
BERT and GPT both use transformers, but BERT is built as an encoder model and GPT as a causal decoder model. Sharing the architecture family does not make their training objectives or runtime behavior the same.