How TTFT works
A TTFT measurement needs two timestamps:
TTFT = time of first output − request start time
The formula is simple. The choice of timestamps is not.
A client-observed TTFT starts when the client sends the request and stops when the client receives the first content-bearing token or streamed chunk. That interval can include:
- Sending the request to the service.
- Authentication, routing, and queueing.
- Turning the prompt into tokens.
- Processing the full prompt, a stage called prefill.
- Selecting the first output token.
- Converting and sending that output back to the client.
During prefill, the model processes the input and prepares the stored attention information it will reuse while generating later tokens. Longer prompts usually require more prefill work. A busy service can add queueing before that work begins.
An engine-level TTFT may use narrower boundaries. For example, vLLM's server metric starts at frontend arrival, currently when tokenization begins. It therefore does not include the client's network path or work done by an upstream application.
This is why TTFT is not one universally comparable number. NVIDIA's benchmarking documentation defines it from query submission to the first token received. Google Cloud's glossary describes it from the model receiving the prompt to producing the first token. Both conventions are useful, but they measure different portions of the request.
Worked example
Suppose one streamed request has this timeline:
- 0 ms: The client sends the prompt and starts its timer.
- 40 ms: The service receives the request.
- 110 ms: Queueing ends and processing starts.
- 350 ms: Prompt processing finishes and the first token is generated.
- 390 ms: The client receives the first content-bearing chunk.
The client-observed TTFT is 390 ms.
A server measurement beginning when the request arrives and ending when the token is generated is 310 ms: 350 ms minus 40 ms.
The 80 ms difference comes from measurement scope. It does not show that either timer is inaccurate.
The same example also shows why you should not read TTFT as total latency. Generation continues after 390 ms. The gaps between later tokens and the time of the final token are separate measurements.
When and why TTFT matters
TTFT matters most when output can be used as soon as it begins. Chat interfaces, coding assistants, and live text systems can show progress after the first content arrives. A shorter TTFT reduces the silent pause before that feedback.
It matters less on its own when the application must wait for the complete answer. A batch summarization job, for example, may care more about total completion time or overall throughput.
Use TTFT to investigate the start of a request. A high value can point to a long prompt, a missed prompt-cache opportunity, queueing under load, a cold worker, frontend overhead, or network distance. TTFT alone cannot tell you which stage is responsible. Pair it with traces or separate queue, prefill, and network measurements.
When comparing systems, hold the workload constant. Use the same prompt-length distribution, cache state, concurrency, streaming behavior, client location, and start and stop events. Report a distribution such as median and tail percentiles rather than one unusually fast request. MLCommons' benchmark methodology uses defined scenarios and treats prompt-phase TTFT separately from generation-phase timing for this reason.
Common misconceptions
A low TTFT does not mean the full response is fast. A system can begin quickly and then generate slowly. Another can begin slowly and produce the remaining output quickly.
Streaming does not automatically reduce model computation time. It lets the client observe partial output before the response is complete. Buffering in a server, proxy, or client can make observed TTFT longer even when the model generates its first token at the same time.
The first chunk is not always one token. Streaming protocols can batch several tokens into one event. Empty events or metadata should not stop the timer if the goal is to measure when usable output appears.
There is no context-free “good” TTFT. Prompt length, load, hardware, caching, network path, and measurement boundaries all change the result. A target should describe the workload and percentile it applies to.
Where to go next
Read What Is Inference Latency? to place TTFT within the full request timeline. Read What Are Tokens per Second? for the generation rate after the first token arrives. To compare the initial wait with generation speed and system capacity, use Latency vs. TTFT vs. Tokens per Second vs. Throughput.