Definition
Sampling is a decoding strategy that selects the next token by drawing from a probability distribution rather than always taking the single highest-probability token. The chosen token is appended to the sequence, the model produces a new distribution for the next position, and the process repeats until generation stops.
Generation systems often transform the distribution before drawing. Temperature changes how concentrated it is. Top-k keeps only a fixed number of candidates. Top-p, also called nucleus sampling, keeps the smallest set whose cumulative probability reaches a threshold. These controls alter which continuations are reachable and how often they are selected; they do not repair an inaccurate model or guarantee creativity.
Origin and usage
Random sampling from probability distributions is much older than language models. Holtzman and colleagues introduced nucleus sampling in a 2019 paper, published at ICLR 2020, after showing that maximum-likelihood decoding could produce bland or repetitive text. Their contribution was the dynamic top-p candidate set, not the general idea of sampling.
Operational significance
Sampling can produce different answers from the same prompt and model. A fixed seed may improve repeatability within a particular stack, but server changes, parallel execution, floating-point behavior, model snapshots, or tool results can still change an end-to-end run. Treat important outputs as probabilistic and test distributions of behavior across repeated trials.
Distinguish it from nearby terms
- Decoding is the broader process of turning model scores into a sequence. Sampling is one decoding family; greedy decoding and beam search need not sample randomly.
- Greedy decoding chooses the highest-scoring token at each step.
- Beam search keeps several high-scoring partial sequences.
- Inference is the complete model execution that produces scores and generated tokens, rather than only the choice rule.
Check your understanding
A team sets top-p to 0.9 and calls the system "10 percent random." Explain why that interpretation is wrong and what the threshold actually controls.