Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

Key points

• Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources.1
• This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL).1
• The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length.1
• We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy.1

Sources (1)

• [1]Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 01:53 AM

  In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability.

  Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources.

Extractive summary: sentences quoted from the sources.