Definition
A decoder-only model places instructions, source material, intermediate text, and generated output into one token sequence. Each position can attend only to earlier positions, so the model learns to predict the next token from the complete prefix. The architecture normally consists of repeated causal self-attention and feed-forward blocks, followed by an output projection over the vocabulary.
The name comes from the decoder side of the Transformer architecture, but it can mislead. The original Transformer decoder also attended to a separate encoder. A decoder-only model removes that encoder and its cross-attention path, keeping the masked self-attention pattern needed for autoregressive generation.
Generative pretraining
OpenAI's 2018 generative-pretraining paper is an early influential example. Alec Radford and colleagues used a multi-layer Transformer decoder for left-to-right language modeling, then fine-tuned it for several language-understanding tasks. Later GPT systems scaled the same general arrangement, but the paper did not establish a universal definition for every decoder-only model.
Distinguish it from nearby terms
An encoder-decoder model gives source and target different architectural paths. A decoder-only model treats them as parts of one prefix and continuation. A causal language model names the training factorization, so causal models can use architectures other than a modern Transformer decoder stack.
Check your understanding
You concatenate an instruction, a retrieved document, and an answer into one training sequence. When predicting the fifth answer token, which earlier material is visible, and why must later answer tokens remain masked?