FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper

Key points

• Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads.1
• It applies CODA's GEMM-plus-epilogue approach to inference, adding direct KV-cache placement, folding the fixed RMSNorm gain into weights, writing FP8 output directly with delayed scaling, and supporting prefill without padding.1
• A block-diagonal Hadamard transform reduces cache quantization error from outlier channels.1
• Across 8B and 70B shapes at batches 1-32, INT4 decode is 1.3-2.4x faster than the faster of PyTorch's BF16 and INT4 blocks at each point: 2.1-3.3x faster than compiled, CUDA-graphed BF16 and 1.3-6.3x faster than the same block using PyTorch's INT4 projection kernel, which targets a single query row and loses to BF16 at batch 32.1

Sources (1)

• [1]FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 05:22 PM

  We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs.

  Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads.

Extractive summary: sentences quoted from the sources.