Definition
A masked language model learns to reconstruct selected tokens that have been hidden or corrupted in an input sequence. Because the model is not restricted to left-to-right attention, a prediction can use visible words before and after the target. This makes the objective useful for learning bidirectional representations.
Masking is a training corruption rule, not necessarily one literal token. The model may see a dedicated mask marker, a random replacement, or an unchanged token selected for prediction. The objective scores only the selected positions under the usual formulation.
BERT's masking recipe
Jacob Devlin and colleagues made masked language modeling central to BERT in 2018. BERT selected 15 percent of token positions for prediction; among those positions, 80 percent were replaced with a mask token, 10 percent with a random token, and 10 percent were left unchanged. Those percentages describe BERT's recipe, not the definition of every masked language model. The paper adopted the term "masked LM" and connected the task to the older Cloze procedure without claiming to invent fill-in-the-blank learning.
Distinguish it from nearby terms
A causal language model predicts each next token from preceding context and directly supports left-to-right generation. A masked language model reconstructs selected positions using bidirectional context. Both are pretraining objectives; neither phrase alone specifies the entire architecture.
Check your understanding
The sentence reads "The build passed after the flaky test was [MASK]." Which context can a masked model use to predict the hidden word, and why can the same objective not generate an answer one future token at a time without modification?