9 Key LLM Inference Metrics Explained in 5 Minutes

Introduction

A high token throughput doesn't necessarily translate to a fast user experience in a large language model (LLM) application. The first token may take too long to arrive, or subsequent tokens may stream too slowly, especially under concurrent load. Throughput alone doesn't capture these differences.

LLM inference metrics help break performance down into measurable components, from the initial response delay to token generation speed and overall system load. This article covers nine key metrics, their formulas, and how to use them to evaluate latency, identify bottlenecks, and track performance over time:

• Per-request latency: time to first token (TTFT), end-to-end latency (E2E), token generation time, time per output token (TPOT), and inter-token latency (ITL)
• Latency across many requests: mean, median, and tail percentiles
• Server load: requests per second (RPS) and tokens per second (TPS)
• Service quality: goodput

We start with the timeline of a single request.

Reading a Request Timeline

Four values define most of the latency metrics below. The value \( t_{\text{send}} \) is the moment the request leaves the client, \( t_{\text{first}} \) is the arrival time of the first output token, \( t_{\text{last}} \) is the arrival time of the last output token, and \( N_{\text{out}} \) is the number of output tokens. ITL also needs the arrival time \( t_j \) of each individual output token, where \( t_1 = t_{\text{first}} \) and \( t_{N_{\text{out}}} = t_{\text{last}} \).

Laid on a timeline, the three timestamps cut a request into two segments:

Before \( t_{\text{first}} \), the request may wait in a queue, and then the server runs prefill, which processes the prompt and produces the first token. After \( t_{\text{first}} \), the server runs decode and emits tokens one step at a time, usually one token per step. Each latency metric below measures one of these segments or the whole span.