What Is Inference Throughput?

It is a measure of system capacity. It does not tell you how quickly one request starts or finishes.

How inference throughput works

The basic calculation is:

throughput = completed work ÷ elapsed time

The word “work” matters. For an image classifier, it might mean images processed. For a language model, it might mean completed requests or tokens processed. A result of “2,000 per second” is meaningless until you know what was counted.

Token throughput needs another label. Input-token throughput measures prompt processing. Output-token throughput measures generated tokens. Total-token throughput combines both. These rates describe different work and should not be compared as if they were the same metric.

The measurement boundary matters too. A result might cover one accelerator, one server, or a whole deployment with many replicas. Dividing a deployment-wide result by its accelerator count can help compare capacity per accelerator, but only when the model and workload are otherwise equivalent.

Throughput usually rises as more requests run concurrently. The serving system can batch work together, using more of the hardware on each processing step. For language models, continuous batching can replace a finished sequence with a waiting one instead of holding the whole batch until its longest sequence ends.

That improvement has a limit. Once compute, memory, or another shared resource is saturated, extra requests wait in a queue. The incoming rate may continue to rise, but the completion rate stops rising with it. Latency then gets worse even if raw throughput changes little.

This is why a useful capacity result is often sustainable throughput under a latency target. The benchmark increases load, measures completed work, and checks that response-time limits still hold. The last passing load is more useful for planning an online service than an unconstrained peak.

A worked example

Suppose a benchmark sends the same mix of prompts to one model server for 60 seconds. The server completes 1,200 requests and generates 180,000 output tokens.

• Request throughput: 1,200 ÷ 60 = 20 requests per second
• Output-token throughput: 180,000 ÷ 60 = 3,000 output tokens per second

Those are two views of the same run. The request rate helps estimate how many interactions the server can finish. The token rate captures differences in response length that the request count hides.

Now raise the offered load to 30 requests per second. The server completes 22 per second, but eight more requests enter the queue each second and the latency target is missed. The offered load is 30 requests per second. The observed completion throughput is 22 requests per second. Neither proves that the service can sustainably handle 30 requests per second.

If 20 requests per second was the highest tested load that kept the queue stable and met the latency target, then 20 is the defensible capacity figure for that workload and target.

Why inference throughput matters

Throughput tells an operator how much hardware is needed for expected demand. It also supports cost measures such as completed requests per dollar or output tokens per accelerator-hour.

For offline jobs, higher throughput can shorten the total processing window. For an online service, enough throughput prevents incoming requests from accumulating faster than the system can finish them.

The target depends on the product. A batch pipeline may accept long waits to maximize total work. An interactive application needs enough throughput while preserving responsive individual requests. A stricter metric called goodput counts only work that also meets the service's latency objectives.

Common misconceptions

Higher throughput means every response is faster

Not necessarily. Batching more requests can raise total output across the server while each request waits longer or receives tokens more slowly. Throughput describes the whole system; latency describes waiting time for an individual request.

Tokens per second always means throughput

“Tokens per second” can describe either aggregate token throughput or the generation rate experienced by one user. Ask whether the number covers one request or all concurrent requests, and whether it counts input, output, or both.

The request rate is the throughput

The request rate is what arrives. Throughput is what completes. They are close only while the system keeps up. If arrivals exceed completions, the queue grows until requests are delayed, rejected, or dropped.

One throughput number is enough to compare systems

Throughput depends on the model, input and output sizes, batch or concurrency pattern, hardware scope, measurement window, and latency limits. Change the workload and the number can change even when the system does not.

Where to go next

Read What Is Inference in AI? for the process whose capacity throughput measures. Compare it with inference latency, which tracks waiting time, and tokens per second, which often describes generation speed. For the complete distinction, see Latency vs. TTFT vs. Tokens per Second vs. Throughput.