Definition
Inference is the execution of a trained or otherwise specified model on input to produce an output. The output may be a score, label, prediction, recommendation, decision, embedding, or generated sequence.
For a classifier, one inference call may map a feature vector to class scores. For an autoregressive language model, inference repeatedly predicts a distribution for the next token, selects or samples a token, appends it to the context, and continues until a stopping condition. One user request may therefore contain many token-level model executions and several application-level calls.
Inference can run online for an interactive request, in batches over stored records, on a device, or through a remote service. The model may be fixed while the runtime changes quantization, batching, kernels, hardware, or numerical precision.
Distinguish it from nearby terms
- Training adjusts model parameters against data and an objective. Inference uses the resulting model to produce outputs.
- Serving is the infrastructure that receives requests, schedules work, executes inference, and returns results. It may add caching, routing, batching, and fallbacks.
- Logical inference derives conclusions under a formal reasoning system. Model inference is the broader operational term and may be statistical or generative.
- An agent run may combine many inference calls with retrieval, tool actions, state changes, and verification.
Operational significance
Measure latency, throughput, memory, hardware, precision, batch policy, decoding settings, and model version. A change in the inference backend can alter cost and, through numerical or scheduling differences, may alter behavior. Reproducibility requires more than a model name.
Check your understanding
A coding agent reads a task, calls a model, searches the repository, calls the model again, edits files, and requests review. Each model execution is inference. The full sequence is an agent run.