Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference

Key points

• While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting impact on energy usage and carbon intensity are scarce.1
• We study quantized inference on Llama 2 7B, Phi 3.5 Mini Instruct, OLMo 1B, Qwen 2.5 7B, and Qwen 2.5 14B across GLUE MNLI, MMLU, HumanEval, and GSM8K datasets, evaluating 10 post-training quantization (PTQ) strategies on NVIDIA H100, A6000, A100, and L4 GPUs.1
• Furthermore, we show that quantization strategies become marginally more energy-efficient relative to full-precision models as batch sizes increase, though gains are modest.1
• We conclude with suggested directions for strategically selecting low-carbon quantization strategies based on specific inference requirements.1

Sources (1)

• [1]Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 12:52 AM

  We present the first unified evaluation of weight-only quantization strategies for LLM inference across varying input/output lengths and GPUs, highlighting differences in energy efficiency in addition to accuracy degradation and runtime.

  While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting impact on energy usage and carbon intensity are scarce.

Extractive summary: sentences quoted from the sources.