During inference, a model might return one result, such as a fraud score, or stream a result piece by piece, such as generated text. A single prediction usually has one obvious completion point. A streamed response does not: the first visible output and the finished response arrive at different times.
How inference latency is measured
Start with the boundary of the measurement.
Model latency usually covers work near or inside the model server. Depending on the platform, it may include preprocessing or communication with a model container.
Service latency can add admission checks, queueing, and batching around the model.
End-to-end latency is measured from the client's point of view. It can include network travel, gateways, service queues, model execution, post-processing, and delivery of the result.
These labels are not perfectly standardized. One platform's “model latency” may include work another platform records as overhead. The timestamps are therefore part of the definition. “From client send to final response received” is comparable; “latency was 800 ms” is not.
For a language model that streams text, the timer can also stop at different output events:
- Time to first token (TTFT) ends when the first output token is received. It captures how soon a response starts to appear.
- End-to-end request latency, sometimes called time to last token, ends when the complete response is received.
- Inter-token latency (ITL) measures the gaps between successive output tokens after generation starts. Some tools call its average time per output token.
Under one common convention:
end-to-end latency = TTFT + generation time
If a response contains n output tokens, its mean inter-token latency after the first token is:
(end-to-end latency - TTFT) / (n - 1)
The denominator is n - 1 because n tokens contain that many gaps after the first token. Check the tool's definition before comparing results. Some tools include the first-token wait in a per-token average, while others exclude it.
A worked example
Suppose a client records these timestamps for a hypothetical streamed response:
- Request sent:
0 ms - First output token received:
450 ms - Final output token received:
1,400 ms - Output length:
20 tokens
The client-observed TTFT is 450 ms. End-to-end latency is 1,400 ms. Generation after the first token takes 950 ms, spread across 19 token-to-token gaps:
950 ms / 19 = 50 ms
The same request therefore has a 450 ms first-token latency, a 50 ms mean inter-token latency, and a 1.40 s completion latency. None of these figures replaces the others.
Now suppose the model server reports 720 ms of model-container work. That does not prove that the remaining 680 ms was network delay. The client and server timers may cover overlapping or differently defined stages. To explain the gap, you need aligned timestamps for queueing, preprocessing, model execution, post-processing, and network transfer.
What changes inference latency
Latency is a result of a whole test, not a permanent property of a model. It changes with:
- Input shape and size. More input can require more preprocessing and model work.
- Output shape and size. A longer generated response takes more sequential generation steps.
- Traffic and concurrency. Requests may wait for capacity when other work is using the same serving resources.
- Batching and scheduling. Grouping requests can improve total system efficiency while making an individual request wait.
- Hardware and serving software. Accelerators, numerical precision, memory movement, kernels, and inference engines affect execution time.
- Warm or cold state. A request can be slower if code, model data, or caches are not ready.
- Application stages. Retrieval, safety checks, tools, and post-processing can add user-visible delay outside the model.
- Network and client location. A server-side benchmark does not include every hop seen by a remote user.
For generated text, input length tends to affect the wait before output begins, while output length strongly affects when the response finishes. Streaming can make the interface feel responsive by revealing partial output earlier, but it does not by itself guarantee a lower completion latency.
Why the average is not enough
A production service handles requests of different sizes under changing load. One average compresses that distribution and can hide the requests with the longest waits.
Latency is often reported with percentiles. A p95 completion latency is the value at or below which 95% of measured requests completed. The remaining 5% took longer. p50 describes the middle request, while p95 or p99 exposes more of the slow tail.
Every reported percentile still needs context. At minimum, label a result with:
- Boundary: client to first token, client to final result, or a named server-side interval.
- Workload: model, input distribution, output distribution, and request settings.
- Load: concurrency or request-arrival pattern.
- Statistic: p50, p95, p99, mean, or a per-request value.
For example, “p95 client-to-final-result latency for this workload at 20 concurrent requests” is a usable measurement. A lower number from a different prompt mix or at single-request load is not evidence of a faster production system.
Why inference latency matters
Latency determines how long one request waits. The relevant stopping point depends on the product.
An interactive interface may care most about first visible output. A background classifier cares about the completed prediction. A multi-step system cannot begin its next dependent step until the prior result is complete, so completion latency can accumulate across the sequence.
Latency also interacts with throughput, which measures how much work a system completes over time. Increasing batch size or concurrency can raise total throughput while increasing the wait for each request. A system can therefore process more total tokens per second and still feel slower to an individual user.
Common misconceptions
“Inference latency” always means model execution time. It does not. The term is also used for service-level and client-observed timing. Look for the measurement boundary.
Time to first token is the response latency. It is one response milestone. A quick first token can be followed by a long generation.
Streaming makes inference finish faster. Streaming changes when partial results are delivered. It can improve perceived responsiveness even if the final result arrives at the same time.
A model has one latency number. The result depends on inputs, outputs, load, hardware, software, and the statistic reported.
Higher total tokens per second means lower latency. Aggregate throughput and per-request waiting time can move in opposite directions.
Where to go next
Use time to first token when the start of a streamed response matters. Use tokens per second to describe generation pace, and inference throughput to describe total serving capacity. For a side-by-side distinction, see latency vs. TTFT vs. tokens per second vs. throughput.