How tokens per second works

The basic calculation is:

TPS = counted tokens / elapsed seconds

During inference, a language model produces tokens as its output. Measuring their rate sounds simple, but a benchmark must make three choices:

  1. Whose tokens? One request or every concurrent request on the system.
  2. Which tokens? Output tokens, input tokens, or both.
  3. Which seconds? Only the generation period after the first token, the full request time, or the full benchmark window.

Those choices create two common meanings.

Per-request output speed describes how fast one response streams after it has started. It is often calculated from the gaps between output tokens, excluding the initial wait. This is the TPS that most closely matches what a person sees while text appears.

System output TPS adds the output tokens from all requests and divides by the benchmark duration. It measures the deployment’s total generation capacity. NVIDIA’s benchmarking definitions and Anyscale’s metrics guide distinguish these two scopes because they can move in opposite directions as load changes.

You may also see input TPS, which measures prompt processing, or total TPS, which combines input and output tokens. Check the label before comparing values. Input processing and output generation are different work, even though both use tokens.

A worked example

Suppose you submit a request at 0.0 s. It produces 41 output tokens:

  • The first token arrives at 0.4 s.
  • The final token arrives at 2.4 s.
  • The 40 intervals after the first token span 2.0 s.

The steady output speed is:

40 intervals / 2.0 seconds = 20 tok/s

A tool that includes the entire request might instead calculate:

41 output tokens / 2.4 seconds = 17.1 tok/s

The two results come from the same response. The first isolates generation after output begins. The second includes the initial wait and uses all output tokens. Current documentation does not use one universal convention: Artificial Analysis excludes the first-token wait from output speed, while some end-to-end metrics include it.

Now change the scope. During a five-second benchmark, several concurrent requests produce 400 output tokens in total:

400 output tokens / 5 seconds = 80 system tok/s

That 80 tok/s is aggregate capacity. It does not say that any one response streamed at 80 tok/s.

Why tokens per second matters

Per-request TPS helps you judge how quickly a long response will arrive once it starts. It matters for chat, code generation, and any workflow that consumes a streamed answer before the answer is complete.

System TPS helps with capacity and cost planning. It shows how much output a deployment can produce under a stated load. Raising concurrency can improve this aggregate rate by using hardware more efficiently, but it can also slow each request. A high-capacity system is not automatically a fast experience for one user.

TPS never tells the whole response-time story. Pair per-request TPS with time to first token, which measures the wait before output begins. Pair system TPS with inference throughput and latency under load. A deployment is useful only if it produces enough work while meeting the response-time needs of its application.

What changes a TPS result

TPS is a property of a measured setup, not just a model name. It can change with:

  • the model’s architecture and size;
  • the hardware and number of devices;
  • the inference engine, numerical precision, caching, and decoding method;
  • prompt and output lengths;
  • concurrency, batching, request rate, and other traffic patterns;
  • the tokenizer used to count tokens; and
  • client-side effects such as streaming chunks and network delivery.

Longer prompts can alter both the initial wait and later output speed. Output speed can also vary with the generated content when the serving engine uses methods such as speculative decoding. A reproducible result therefore names the model and endpoint, workload shape, load, tokenizer, and exact formula. vLLM’s benchmark documentation recommends comparing measurement points and formulas rather than metric names alone.

Common misconceptions

“TPS tells me how soon the answer starts”

TPS usually describes the rate during generation or across a benchmark window. It does not isolate the blank wait before the first token. Two systems can have the same output TPS and very different time to first token.

“More system TPS means a faster response for me”

Not necessarily. Serving more requests together can raise total system TPS while each request receives tokens less often. Ask whether a result is per request or aggregate.

“A token is a word”

Tokens are not fixed units of visible text. Different tokenizers can split the same sentence differently, so equal tok/s values can correspond to different amounts of text. Cross-model benchmarks sometimes retokenize outputs with a common tokenizer for this reason.

“TPS is one standardized metric”

There is no single enforced definition. TPS may exclude or include the first-token wait, and it may count output tokens alone or input and output together. Always read the formula.

Where to go next

Use inference latency to understand response delay, time to first token for the initial wait, and inference throughput for total serving capacity. For the boundaries between all four measurements, see Latency vs. TTFT vs. Tokens per Second vs. Throughput.