Inference and generation

Beam search

A decoding algorithm that keeps a fixed number of high-scoring partial sequences while generating output.

Chain-of-thought prompting

Prompting a model to produce or use intermediate reasoning steps before an answer.

Continuous batching

An inference scheduling technique that adds and removes generation requests at iteration boundaries as capacity becomes available.

Few-shot prompting

Supplying a small set of worked examples in context to steer task behavior without updating model weights.

Greedy decoding

Generating each next token by selecting the current highest-probability candidate.

In-context learning

A model's ability to adapt behavior from instructions, examples, or patterns supplied within the current context without parameter updates.

Input token

A token supplied to a model for an inference call, including user content and any instructions, history, retrieved material, tool definitions, or other context assembled by the system.

KV cache

Stored attention keys and values from earlier tokens that an autoregressive transformer reuses instead of recomputing them for every new token.

Logit

An unnormalized score produced by a model before conversion into probabilities.

Maximum output tokens

A hard limit on how many tokens a model may generate in one response.

One-shot prompting

Supplying one worked example in context to demonstrate the desired task or output pattern.

Output token

A token generated by a model as part of its response, often metered separately from input tokens.

Prompt

Input supplied to a model to condition the output, including instructions, examples, context, and user data.

Prompt chaining

Connecting multiple model calls so one call's structured output becomes context or input for a later call.

Sampling

Selecting an output token from the probability distribution produced by a generative model.

Stop sequence

A configured token or text pattern that causes generation to terminate when produced.

Temperature

An inference setting that reshapes token probabilities, with higher values generally increasing variation and lower values concentrating choices.

Top-p sampling

A decoding method that samples only from the smallest set of candidate tokens whose cumulative probability reaches a chosen threshold.

Zero-shot learning

Performing a task or recognizing a category without task-specific labeled examples supplied for that use.

Context engineering

Designing how instructions, state, knowledge, examples, tools, and feedback are selected, structured, and delivered to a model at the moment they are needed.

Function calling

A model interface in which the model selects a named function and supplies structured arguments for application code to execute.

Grounding

Connecting an AI output to identifiable evidence, data, observations, or constraints outside the model's unsupported generation.

Hallucination

An output that presents unsupported or incorrect content as though it were grounded or factual.

Prompt engineering

Designing and testing model inputs to elicit useful behavior from a particular model and task.

Reasoning model

A model optimized to spend additional inference effort on multi-step problem solving before returning an answer or action.

Reasoning token

A provider-reported token used by a reasoning model for intermediate inference work before or alongside its visible answer.

Structured output

Model output constrained to a machine-readable schema such as JSON Schema so downstream software can validate and consume it reliably.

System prompt

High-priority runtime instructions supplied by an application to establish the model's role, constraints, and operating context.

Test-time compute

Additional computation spent during inference, such as longer deliberation, search, candidate generation, or verification, to improve an outcome.