Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving

Key points

• Large Language Model (LLM) inference is driving a growing share of datacenter carbon emissions.1
• While prior studies investigate the carbon savings achievable through spatial scheduling, they oversimplify the dynamics of modern inference engines, such as queueing and continuous batching, which strongly shape inference latency.1
• We address this issue by designing and evaluating a latency-centric, geo-distributed inference simulator, calibrated with hardware-grounded latency and power measurements, that reproduces these dynamics and enables the study of carbon savings under strict service-level objectives (SLOs).1
• Our characterization further shows that savings grow with the spread of carbon intensity across datacenters and shrink as system load increases, while SLO attainment depends on how frequently schedulers receive datacenters' load updates.1

Sources (1)

• [1]Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 04:38 AM

  Although carbon-aware scheduling can reduce these emissions in geo-distributed datacenters by exploiting the spatiotemporal variations in grid carbon intensity, the latency-sensitive nature of LLM serving constrains this design space.

  Large Language Model (LLM) inference is driving a growing share of datacenter carbon emissions.

Extractive summary: sentences quoted from the sources.