Definition
Latency is the elapsed time between a defined starting event and a defined response or completion point. The endpoints must be stated. A user may care about time to first visible output, time to a useful answer, or time to completed side effects. An operator may separately measure queue delay, model processing, tool calls, retries, validation, and promotion.
For streaming language models, time to first token measures initial responsiveness while inter-token latency or generation rate describes the rest of the stream. Neither equals end-to-end task latency when an agent must search, call tools, wait for external systems, run tests, or obtain approval.
Report a distribution
Average latency hides the slow runs that often determine user experience and system capacity. Report percentiles such as p50, p95, and p99 over a named workload and time window. Include timeouts, cancellations, and failed runs in a clearly defined way rather than silently dropping them.
Distinguish it from nearby terms
Throughput measures completed work per unit time. Concurrency counts work in progress. Service time is active processing; response time can include queueing and waits. A batched system may improve throughput while making one request wait longer, so the metrics can move in opposite directions.
Check your understanding
An agent responds "working on it" in 300 milliseconds, then completes the task in twelve minutes. Is its latency 300 milliseconds? Only for the acknowledgement. Report first response and completed, verified outcome separately, with queueing, model, tool, and approval time when diagnosing the delay.