Inference and generation

Beam search

A bounded search algorithm that keeps a fixed number of high-scoring partial sequences at each decoding step.

Chain-of-thought prompting

Prompting a model with worked intermediate reasoning, or asking it to generate intermediate steps, before producing an answer.

Context engineering

Designing how instructions, state, knowledge, examples, tools, and feedback are selected, structured, and delivered to a model at the moment they are needed.

Continuous batching

An inference scheduling technique that adds and removes generation requests at iteration boundaries as capacity becomes available.

Few-shot prompting

Supplying a small set of worked examples in context to steer task behavior without updating model weights.

Function calling

A model interface in which the model selects a named function and supplies structured arguments for application code to execute.

Greedy decoding

Generating each next token by selecting the current highest-probability candidate.

Grounding

Connecting an AI output to identifiable evidence, data, observations, or constraints that support its claims.

Hallucination

Generated content that is unsupported by the required source or context, conflicts with it, or presents invented material as factual.

In-context learning

A model's ability to adapt behavior from instructions, examples, or patterns supplied within the current context without parameter updates.

Input token

A token supplied to a model for an inference call, including user content and any instructions, history, retrieved material, tool definitions, or other context assembled by the system.

KV cache

Stored attention keys and values from earlier tokens that an autoregressive transformer reuses instead of recomputing them for every new token.

Logit

An unnormalized model score supplied to a normalization function such as softmax; in statistics, logit also means the logarithm of odds.

Maximum output tokens

A request-level ceiling on the tokens a model may generate, which can include hidden reasoning as well as visible output on some APIs.

One-shot prompting

Supplying one worked example in context to demonstrate the desired task or output pattern.

Output token

A token generated by a model as part of its response, often metered separately from input tokens.

Prompt

Input supplied to a model to condition the output, including instructions, examples, context, and user data.

Prompt chaining

Connecting multiple model calls so one stage's result becomes context or input for a later stage.

Prompt engineering

Designing and testing model inputs to elicit useful behavior from a particular model and task.

Reasoning model

A model optimized to spend additional inference effort on multi-step problem solving before returning an answer or action.

Reasoning token

A provider-reported token used by a reasoning model for intermediate inference work before or alongside its visible answer.

Sampling

Selecting an output token from the probability distribution produced by a generative model.

Stop sequence

A configured token or text pattern that causes generation to terminate when produced.

Structured output

Model output constrained to a machine-readable schema such as JSON Schema so downstream software can validate and consume it reliably.

System prompt

A high-authority runtime instruction layer that sets a model's role, constraints, tools, and operating context before it handles user input.

Temperature

A decoding setting that rescales token logits before sampling, making the next-token distribution sharper or flatter.

Test-time compute

Computation spent after a request arrives on reasoning, search, candidate generation, tool use, or verification before returning an answer.

Top-p sampling

A decoding method that samples only from the smallest set of candidate tokens whose cumulative probability reaches a chosen threshold.

Zero-shot learning

Recognizing unseen classes through shared semantics, or, in current LLM usage, attempting a task without demonstrations in the prompt.