The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token

Key points

• Energy-efficient deployment of large language models requires understanding how model architecture, hardware, and serving settings shape energy per generated token.1
• It relates token energy to two core dimensions: active bytes and realized decode time.1
• Active bytes describe logical byte demand per output, separating dense weights shared within a decode iteration from private key--value (KV) histories read and updated for each request.1
• Our experiments cover three models under BF16 decoding on NVIDIA Hopper and Blackwell GPUs, including two-GPU tensor parallelism on Hopper.1

Sources (1)

• [1]The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 05:32 PM

  We introduce the Token-Energy Law, a compact, interpretable model of GPU-board energy per output token for dense, full-history decoding.

  Energy-efficient deployment of large language models requires understanding how model architecture, hardware, and serving settings shape energy per generated token.

Extractive summary: sentences quoted from the sources.