Definition
A probability distribution is a rule that assigns probability to possible outcomes or sets of outcomes. Every probability is nonnegative, and the probability of the entire outcome space is one.
For a discrete variable, a probability mass function assigns probability to each outcome, and those values sum to one. A language model's next-token distribution is discrete because it allocates probability across a finite vocabulary. For a continuous variable, a probability density describes relative concentration. The density integrates to one across its range, but the probability of any exact point is usually zero. Probabilities come from areas over intervals, not from treating density values as point probabilities.
A distribution can describe observed data, a model's prediction, a prior assumption, or uncertainty about an unknown quantity. Its shape matters. Two distributions can have the same mean while assigning very different probability to extremes.
Distinguish it from nearby terms
- Logits are raw model scores. A function such as softmax can convert them into a normalized discrete distribution.
- A sample is one outcome drawn according to a distribution. It is not the distribution itself.
- A dataset distribution describes how examples occur in data. A model's predictive distribution describes its uncertainty over outputs for an input.
- A confidence score may look like a probability without being calibrated. Summing to one does not prove that predicted frequencies match reality.
Operational significance
Decision rules often depend on more than the most likely outcome. Expected loss, tail risk, and calibration can change which action is sensible. For generated text, decoding changes how the next-token distribution becomes an output. Greedy decoding chooses the highest-probability token, while sampling can choose lower-probability tokens according to a defined rule.
Check your understanding
A forecast assigns 60 percent to rain and 40 percent to no rain. Rain is the most likely single outcome, but one dry day does not disprove the distribution. Calibration must be checked across many comparable forecasts.