The performance model of LLM serving, split into distinct metrics: TTFT (time to first token, dominated by the prefill of the prompt), TPOT/ITL (inter-token latency during decode), end-to-end latency, and throughput (total tokens/sec across all users).
How it works
Inference in large language models (LLMs) occurs in two distinct phases: Prefill and decode. During the prefill phase, the model processes the entire prompt in parallel, which is compute-bound and sets the time to first token (TTFT). This means that the time it takes to get the first response is primarily determined by how quickly the model can handle the entire input. In contrast, the decode phase is memory-bandwidth-bound, where the model generates one token at a time. This phase determines the throughput, or the number of tokens generated per second. The interplay between these two phases is crucial for performance; for instance, larger batches can improve throughput by amortizing the computational load across multiple requests, but they also increase the time each request spends in the queue, which can negatively impact TTFT.
the length of the prompt affects TTFT significantly. Longer prompts require more time to process, thus inflating the TTFT. Similarly, if the generated output is lengthy, it can extend the total time taken for a response. This means that while optimizing for throughput, one must also consider how it impacts the latency experienced by users. The challenge lies in balancing these metrics to ensure a responsive experience while maximizing the overall efficiency of the system.
To illustrate, consider a scenario where a model is set to handle both interactive chat and bulk processing. In the chat scenario, a user expects a quick response, so the focus should be on minimizing TTFT, even if that means sacrificing some throughput. Conversely, for bulk processing tasks, the goal shifts to maximizing throughput, where longer wait times are acceptable in exchange for processing many tokens efficiently. Understanding these dynamics helps in configuring the model to meet specific user needs.
- Prefill phase sets TTFT.
- Decode phase sets tokens/sec.
- Bigger batches increase throughput.
- Long prompts inflate TTFT.
- Long outputs inflate total time.
The software-engineering analogue
In software engineering, we often deal with similar concepts of latency and throughput, particularly in systems involving queues and resource management. For example, one might apply Little's Law to analyze the relationship between the average number of items in a queue, the average time they spend there, and the rate at which they are processed. In this context, adjusting the batch size in a job queue can help balance the load and optimize performance.
However, the analogy breaks down when we consider that in LLMs, throughput is limited not just by the queue but also by the GPU memory bandwidth and the size of the key-value (KV) cache. Unlike traditional systems where a single adjustment can often optimize both latency and throughput, in LLMs, these two phases have opposing bottlenecks. This means that tuning one aspect can inadvertently worsen the other, making it essential to treat TTFT and throughput as separate metrics that require distinct strategies for optimization.
Why not the obvious alternative
A common initial approach to improving performance might be to optimize a single average-latency metric. This can seem straightforward, but it often obscures the underlying tradeoffs between TTFT and throughput. For instance, if the batch size is too small, the GPUs may remain idle, leading to inefficient resource use. On the other hand, if the batch size is too large, users may experience frustrating delays, resulting in a poor experience.
By separating TTFT from inter-token latency (TPOT), engineers can more effectively tune each metric against its specific bottleneck. This targeted approach prevents the pitfalls of a one-size-fits-all strategy, allowing for a more nuanced understanding of how to balance user experience with system efficiency. Without this separation, teams risk making decisions that could lead to either wasted computational resources or dissatisfied users.
When to use which
When deciding how to configure an LLM system, the use case plays a critical role. For interactive chat applications, the priority should be on minimizing TTFT and ensuring that the 95th percentile latency (p95) remains low. This often means capping batch sizes and prioritizing the prefill phase to ensure quick responses. In contrast, for bulk or offline processing tasks, the focus shifts to maximizing throughput. Here, larger batch sizes are acceptable, and users can tolerate higher latencies in exchange for improved overall processing efficiency. In mixed environments, strategies like prefill and decode disaggregation or priority scheduling can be employed to balance the needs of both interactive and batch processing.
In production: when it breaks
Why it happens. This latency increase can occur due to inefficient batching of requests. When the model receives requests that are not optimally batched, it can lead to higher TTFT as the system spends more time pre-filling prompts rather than processing multiple requests simultaneously.