What is Latency?
Also called response time, wall clock time.
Latency is the elapsed time between a request being issued and its response being complete. It is reported as a distribution rather than a single number, usually with percentiles such as the median, the ninety-fifth, and the ninety-ninth. In agent systems the total is dominated by the number of sequential model and tool calls rather than by raw computation.
Latency is measured as a distribution because request durations vary widely. Percentiles describe it: a median shows the typical experience, while the ninety-fifth and ninety-ninth percentiles describe the slow tail that produces timeouts and complaints. Where the measurement starts and stops matters too, since a server-side number excludes queueing, network time, and client rendering that the user experiences as part of the wait.
In systems built on model calls, total time is usually dominated by the number of sequential steps rather than by how fast any one step is. Each additional tool call, retrieval, or verification pass adds a full round trip. That is why running independent steps in parallel, cutting unnecessary steps, and streaming partial output often help more than optimizing individual calls.
Averages are the classic trap, because one run in a hundred taking two minutes barely moves a mean while defining the user experience for that segment. Another trap is a feedback loop: slow requests hold resources, queues build, and latency worsens for everyone, so a tail problem becomes a capacity problem. Timeouts set below the real tail convert slowness into failures and retries.
Latency relates to throughput and concurrency through a simple identity: the amount of work in progress equals throughput multiplied by latency. Raising concurrency to fix latency only helps when the constraint is the local processing tier. Time to first token is a separate, narrower measure that captures perceived responsiveness for streaming interfaces, and the two can move in opposite directions.
Key points
- Always a distribution; report percentiles, never only the average.
- Tail latency drives timeouts, retries, and user abandonment.
- Agent latency scales with sequential steps more than with model speed.
- Measure at the client boundary to include queueing and network time.
In practice
A document review agent shows a median latency of nine seconds, which looks fine. The ninety-ninth percentile is one hundred and ten seconds, because long documents trigger an extra chunking pass and a second model call. Those slow runs hit the client timeout of sixty seconds and are retried, doubling load. Raising the timeout and splitting the pass fixes both problems.