Definition
Top-p sampling sorts candidate tokens from most to least probable, keeps the smallest leading set whose cumulative probability reaches a threshold p, renormalizes that set, and samples from it. The number of eligible tokens changes at every generation step. A peaked distribution may need only a few tokens to reach p, while a flatter one may need many.
A setting of p = 1 keeps the full distribution, subject to any other filters. Lower values remove more of the low-probability tail. Top-p does not mean "keep the top p percent of tokens," nor does it keep every token whose individual probability exceeds p.
Ari Holtzman and coauthors introduced the method as nucleus sampling in their 2019 paper, later published at ICLR 2020. "Top-p" became the common parameter name in generation APIs.
Distinguish it from nearby terms
- Temperature reshapes relative probabilities. Top-p chooses a variable-sized candidate set based on cumulative mass.
- Top-k sampling always keeps a fixed number of candidates. Top-p keeps as many as the current distribution requires.
- Greedy decoding does not sample from a candidate set; it picks the highest-probability token.
Check your understanding
A decoding configuration uses top_p = 0.9. Does that mean every eligible token has at least 90 percent probability? Explain what the threshold actually controls.