---
title: 'Causal language model'
description: 'A language model trained to predict each token from the tokens that precede it, without access to future positions.'
canonical_url: 'https://darkfactory.dev/glossary/causal-language-model'
markdown_url: 'https://darkfactory.dev/glossary/causal-language-model.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Causal language model


## Definition

A causal language model assigns a sequence probability one token at a time. For tokens x1 through xn, it models each xt using only x1 through x(t-1). During Transformer training, a causal attention mask blocks each position from reading later tokens. The training system can still calculate losses for all positions in parallel because the complete training sequence is already known.

At generation time, the model samples or selects one token, appends it to the context, and repeats. Generation is therefore sequential even when training is highly parallel. "Causal" describes the direction of information flow in the sequence. It does not mean the model performs causal inference about interventions in the world.

## Generative pretraining

OpenAI's 2018 generative pretraining paper used a Transformer language model with this left-to-right objective, then fine-tuned the same model for downstream tasks. It is an influential decoder-style example, not the origin of next-token language modeling.

## Distinguish it from nearby terms

A masked language model predicts hidden positions using context on both sides. A causal language model cannot inspect future positions. "Autoregressive" is broader: it describes any model that factors an output into conditional steps, including models outside language.

## Check your understanding

A training example contains "the build is green." When the model is scored on "green," which tokens may contribute to that prediction, and which token must the mask hide?

## Related terms

- [Masked language model](https://darkfactory.dev/glossary/masked-language-model)
- [Decoder-only model](https://darkfactory.dev/glossary/decoder-only-model)
- [Token](https://darkfactory.dev/glossary/token)
- [Pretraining](https://darkfactory.dev/glossary/pretraining)

## Evidence and further reading

- [Google Machine Learning Glossary](https://developers.google.com/machine-learning/glossary/)
- [Improving Language Understanding by Generative Pre-Training](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)