Inference & Serving

What happens when a model runs: inference, latency, throughput, batching and the serving infrastructure behind every AI product.

Learn

Inference & Serving

ComparisonLatency vs. TTFT vs. Tokens per Second vs. ThroughputLatency is how long one inference request takes, TTFT is how long a streamed request takes to begin answering, tokens per second is a rate that may describe one stream or a whole system, and throughput is how much total work a system completes over time. The sharpest distinction is scope: latency and TTFT time a request, while throughput counts work across a measurement window.
ConceptWhat Are Tokens per Second?Tokens per second (TPS or tok/s) measures how many tokens an AI system produces in one second. The number may describe the output speed of one response or the combined throughput of an entire serving system, so the label alone is incomplete.
ConceptWhat Is Inference Latency?Inference latency is the elapsed time between an AI system receiving an input and producing a specified output. A latency number is meaningful only when you know where the timer starts, where it stops, and what kind of output counts as complete.
ConceptWhat Is Inference Throughput?Inference throughput is the amount of model work an inference system completes per unit of time across all the requests it is serving. It may be measured in completed requests, inferences, samples, or tokens per second.
ConceptWhat Is Inference in AI?Inference in AI is the process of running a trained model on an input to produce an output, without updating what the model learned. It is the use phase of a model: a photo goes in and a label comes out, or a prompt goes in and generated text comes out.
ConceptWhat Is Time to First Token?Time to first token, or TTFT, is the elapsed time between starting a language-model request and getting the first token of its response. It measures the initial wait, not how fast the rest of the response arrives.