The phrase “tokens per second” needs special care. It names a unit, not a complete metric. You still need to know whose tokens were counted and which time interval was used.
What each metric measures
Inference latency
Inference latency is elapsed time for a request. For a generated response, “end-to-end latency” normally runs from sending or receiving the request to receiving its final output.
It answers: How long until this request is done?
That boundary is not universal. A client-side measurement can include routing, network transfer, queueing, and response handling. An inference server may start its clock after some of those steps. A benchmark that says only “latency” has not told you enough to reproduce the number.
Latency also depends directly on output length. A concise and a long response can have the same TTFT and generation pace, but the long response takes more time to finish.
Time to first token
Time to first token, or TTFT, runs from the request boundary to the first non-empty output token in a streamed response.
It answers: How long until the answer starts appearing?
TTFT commonly includes queueing, input processing, the model’s prefill work over the prompt, generation of the first token, and whatever network time falls inside the chosen boundary. It is part of end-to-end latency, not an alternative name for it.
A low TTFT makes an interface feel responsive. It says nothing by itself about how smoothly the rest of the response will arrive or when that response will finish.
Tokens per second
Tokens per second is an output-token count divided by elapsed seconds. In product comparisons, it often means one request’s generation speed after the first token.
It answers: How quickly does generated text arrive once generation is underway?
For a response with (N) output tokens, there are (N-1) intervals between the first and last token. If those intervals take (G) seconds, the post-first-token rate is:
[
\text{per-request generation rate} = \frac{N-1}{G}
]
This is the reciprocal of average time per output-token interval when each measured event contains one token. Real streaming APIs may deliver chunks containing multiple tokens, so a client must count tokens rather than assume one token per event.
Some tools instead divide all output tokens by the request’s full end-to-end latency. Some combine tokens from every concurrent request. Those choices produce different numbers, even though all may be labeled TPS or tokens/s.
Inference throughput
Inference throughput is the total amount of work a system completes per unit of time under a stated workload. It can be measured in output tokens per second, total tokens per second, requests per second, samples per second, or another workload-appropriate unit.
It answers: How much load can the system handle?
Throughput is an aggregate property. It needs a test interval and a load pattern. Concurrency, request arrival rate, prompt and response lengths, batching, replica count, and failed-request handling can all change it.
Request throughput and token throughput are not interchangeable. A server can complete many short requests or fewer long requests while producing the same number of output tokens per second.
The actual distinction
| Metric | Clock stops at | Typical scope | Typical unit | The question it answers |
|---|---|---|---|---|
| Latency | Final response output | One request | ms or s/request | When is this request finished? |
| TTFT | First content token | One streamed request | ms or s/request | When does the answer begin? |
| Tokens per second | Depends on the stated formula | One request or the whole system | output tokens/s | How fast are tokens arriving, and for whom? |
| Throughput | End of a test interval | System or deployment | tokens/s, requests/s, or samples/s | How much work is completed under load? |
Three questions disambiguate almost any reported result:
- What is counted? Output tokens, all tokens, completed requests, or samples?
- Where does the clock start and stop? At client send, server receipt, first token, final token, or fixed test boundaries?
- What is the scope? One request, one user, one replica, or the complete deployment under concurrent load?
If a benchmark cannot answer all three, its number is not ready for comparison.
A worked example
Imagine one request that returns 81 output tokens:
- The request is sent at 0.0 seconds.
- The first token arrives at 0.8 seconds.
- The final token arrives at 2.8 seconds.
Its TTFT is 0.8 seconds. Its end-to-end latency is 2.8 seconds. Generation after the first token spans 2.0 seconds and contains 80 token-to-token intervals, so its post-first-token rate is 40 tokens per second.
If a tool counts all 81 output tokens and divides by the full 2.8-second latency, it reports about 29 tokens per second. That is not a mathematical disagreement. It is a different clock boundary and numerator.
Now place many requests on one server. During a 10-second test window, the server generates 3,000 output tokens and completes 50 requests:
- Output-token throughput: (3{,}000 / 10 = 300) tokens per second
- Request throughput: (50 / 10 = 5) requests per second
The server’s 300 tokens per second is the combined output across active requests. An individual user may still receive output at 40 tokens per second, or less if added load increases queueing and decode delays.
When each metric matters
For a chat interface, coding assistant, or other streamed interaction, TTFT describes the initial pause. Per-request generation rate, or its inverse time per output token, describes the pace after that pause. End-to-end latency matters when the user or another program needs the complete answer before continuing.
For a batch job, throughput usually matters more than TTFT. The job cares about total completion rate, provided failures and output quality remain acceptable.
For capacity planning, use throughput together with a latency objective. Increasing concurrency often improves aggregate throughput at first because the hardware does more work in parallel. The same concurrency can increase queueing and slow each user. After saturation, throughput may flatten or fall while latency keeps rising.
For a non-streaming API, TTFT may not be observable at the client because the response is delivered only when complete. The server can still measure internal stages, but client-visible performance is end-to-end latency.
What people confuse and why
Low TTFT does not guarantee low latency
A response can start quickly and then generate slowly. TTFT captures only the first milestone. Total latency also includes the remaining generation time.
Under an explicit post-first-token definition, the relationship is:
[
\text{end-to-end latency} = \text{TTFT} + (N-1)\times\text{average time per output-token interval}
]
Implementations use different counting conventions, so check the benchmark’s formula before reconstructing one metric from another.
Tokens per second can be throughput
“Tokens per second” sometimes means one user’s generation pace. It can also be the unit for aggregate output-token throughput. The label alone cannot distinguish them.
Look for qualifiers such as per request, per user, per stream, aggregate, or system. If none appears, inspect the formula and test setup.
High throughput does not guarantee fast requests
A server can batch many requests to produce more total tokens each second while each request waits longer or receives tokens less often. This is why serious serving tests report throughput alongside latency percentiles or calculate goodput: work completed while meeting a stated service target.
Unlike workloads produce unlike results
TTFT changes with prompt length and queueing. End-to-end latency changes with response length. Throughput changes with concurrency, arrival pattern, batching, and hardware allocation. Token rates also depend on the tokenizer used to count output.
Compare systems only with the same model behavior, prompt and output distributions, load pattern, streaming mode, sampling settings, tokenizer, measurement boundary, and success criteria. A single idle request tests something different from a saturated server.
Averages hide the requests users remember
An average TTFT or latency can look healthy while a small share of requests wait much longer in a queue. Report a distribution, including relevant tail percentiles, and include the number of observations. For throughput, report errors and rejected requests rather than counting only successful work without saying so.
How to read an inference benchmark
Before trusting a chart or model-card claim, check that it states:
- whether tokens per second is per request or aggregate;
- whether it counts output tokens only or input plus output tokens;
- whether TTFT or the first token is included in the token-rate calculation;
- whether latency is client-side, server-side, or model-only;
- prompt-length and output-length distributions;
- concurrency and request arrival pattern;
- warm-up, test duration, and start/end boundary handling;
- mean and tail latency, not just one average;
- successful-request and error criteria;
- model, precision, hardware, engine, and replica count.
The best metric is not one of these four in isolation. It is the metric that matches the user-visible or operational goal, measured at realistic load with every boundary stated.
Where to go next
Read What Is Inference Latency? for request timing boundaries, What Is Time to First Token? for the initial streamed-response delay, What Are Tokens per Second? for token-rate formulas, and What Is Inference Throughput? for system capacity under load.