Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware

Key points

• Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability.1
• We directly measure this trade-off for 86 open-weight model checkpoints, evaluated on seven benchmarks and six hardware platforms ranging from a consumer GPU to an eight-GPU server.1
• Most configurations lie away from the score--energy Pareto frontier: they use a median of 2.6--4.8$\times$ more energy than another configuration that scores at least as well, with differences reaching 27$\times$.1
• Reasoning can improve capability, but enabling thinking increases energy by about 3.7$\times$ on average, and the benefit depends strongly on the model family and output budget.1

Sources (1)

• [1]Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 11:15 PM

  We find large differences in energy efficiency even among models with similar capability.

  Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability.

Extractive summary: sentences quoted from the sources.