Definition
Greedy decoding selects the highest-scoring next token at every generation step and commits to that choice. In a language model, the selection normally happens after the system has applied any configured constraints or score adjustments to the output logits.
The method is locally optimal. It chooses the best token for the current prefix, then treats that choice as fixed. A lower-ranked token cannot be reconsidered even if it would have opened a better complete sequence. Greedy decoding is therefore fast and easy to reproduce under a fixed implementation, but it is not a search for the globally highest-scoring sequence and it is not a guarantee of human-preferred text.
Determinism and its limits
Greedy decoding contains no random sampling step. That makes it deterministic for identical logits and an identical implementation. It does not guarantee byte-for-byte agreement across serving stacks. Tokenization, chat templates, numerical precision, kernels, tie handling, and score processors can change the logits or the selected token.
A 2026 controlled preprint by Shahed Masoudian and coauthors found measurable backend differences even under greedy settings. Its exact effect sizes remain provisional. Reproducing such an evaluation requires the model checkpoint, backend and version, prompt rendering, and full generation configuration.
Distinguish it from nearby terms
- Sampling draws from a probability distribution and introduces a random choice unless the implementation fixes all relevant randomness.
- Beam search retains several partial sequences. Greedy decoding retains one.
- Temperature changes the token distribution used for sampling. Setting temperature to zero is often an API convention for greedy behavior, not the mathematical definition of the algorithm.
Check your understanding
At the first step, token A scores slightly above token B, but every continuation after B scores much higher than every continuation after A. Explain which token greedy decoding selects and why it cannot recover.