TTFT 的工作原理
测量 TTFT 需要两个时间戳：
TTFT = 首个输出的时间 − 请求开始时间
公式很简单，但时间戳的选择并不简单。
从客户端观察到的 TTFT，从客户端发送请求开始，到客户端收到首个承载内容的令牌或流式数据块为止。这个时间间隔可能包括：
- 将请求发送到服务。
- 身份验证、路由和排队。
- 将提示转换为令牌。
- 处理完整提示，这一阶段称为预填充（prefill）。
- 选择首个输出令牌。
- 转换输出并将其发送回客户端。
在预填充阶段，模型会处理输入，并准备好在生成后续令牌时重复使用的注意力信息。提示越长，通常需要的预填充工作就越多。服务繁忙时，相关工作开始前还可能发生排队。
引擎级 TTFT 可能采用更窄的计时边界。例如，vLLM 的服务器指标从前端收到请求时开始，目前具体是在开始分词时计时。因此，它不包括客户端的网络路径，也不包括上游应用执行的工作。
因此，TTFT 并不是一个在所有情况下都具有普遍可比性的数值。NVIDIA 的基准测试文档将其定义为从提交查询到收到首个令牌之间的时间。Google Cloud 的术语表则将其描述为从模型收到提示到生成首个令牌之间的时间。这两种约定都很有用，但它们测量的是请求中不同的部分。
示例
假设某个流式请求的时间线如下：
- 0 毫秒：客户端发送提示并启动计时器。
- 40 毫秒：服务收到请求。
- 110 毫秒：排队结束，处理开始。
- 350 毫秒：提示处理完成，并生成首个令牌。
- 390 毫秒：客户端收到首个承载内容的数据块。
从客户端观察到的 TTFT 是390 毫秒.
A server measurement beginning when the request arrives and ending when the token is generated is 310 ms: 350 ms minus 40 ms.
The 80 ms difference comes from measurement scope. It does not show that either timer is inaccurate.
The same example also shows why you should not read TTFT as total latency. Generation continues after 390 ms. The gaps between later tokens and the time of the final token are separate measurements.
When and why TTFT matters
TTFT matters most when output can be used as soon as it begins. Chat interfaces, coding assistants, and live text systems can show progress after the first content arrives. A shorter TTFT reduces the silent pause before that feedback.
It matters less on its own when the application must wait for the complete answer. A batch summarization job, for example, may care more about total completion time or overall throughput.
Use TTFT to investigate the start of a request. A high value can point to a long prompt, a missed prompt-cache opportunity, queueing under load, a cold worker, frontend overhead, or network distance. TTFT alone cannot tell you which stage is responsible. Pair it with traces or separate queue, prefill, and network measurements.
When comparing systems, hold the workload constant. Use the same prompt-length distribution, cache state, concurrency, streaming behavior, client location, and start and stop events. Report a distribution such as median and tail percentiles rather than one unusually fast request. MLCommons' benchmark methodology uses defined scenarios and treats prompt-phase TTFT separately from generation-phase timing for this reason.
Common misconceptions
A low TTFT does not mean the full response is fast. A system can begin quickly and then generate slowly. Another can begin slowly and produce the remaining output quickly.
Streaming does not automatically reduce model computation time. It lets the client observe partial output before the response is complete. Buffering in a server, proxy, or client can make observed TTFT longer even when the model generates its first token at the same time.
The first chunk is not always one token. Streaming protocols can batch several tokens into one event. Empty events or metadata should not stop the timer if the goal is to measure when usable output appears.
There is no context-free “good” TTFT. Prompt length, load, hardware, caching, network path, and measurement boundaries all change the result. A target should describe the workload and percentile it applies to.
Where to go next
Read What Is Inference Latency? to place TTFT within the full request timeline. Read What Are Tokens per Second? for the generation rate after the first token arrives. To compare the initial wait with generation speed and system capacity, use Latency vs. TTFT vs. Tokens per Second vs. Throughput.