Is 2:4 Sparsity Still Worth It? Semi-Structured Sparsity versus Quantization in LLM Inference

Key points

• When the two are combined, the practical question is which step to take next: from a given precision, add 2:4 sparsity or drop to the next lower precision?1
• We compare the two at equal bits, with sparsity metadata counted, and at equal nominal tensor-core tiers, over all six GEMM operands of a transformer layer, with kernels measured on three GPU generations and online pruning and quantization included.1
• Our results show that as long as a lower precision is available, adding 2:4 is the worse step, both less accurate and slower, because the energy that 2:4 discards from each group of four along the hardware-forced axis does not shrink with bits, while quantization error does.1
• Once quantization reaches the lowest precision the hardware supports, 2:4 still speeds up linear layers by 1.3--1.7\x{}, and on an H100 a retrained 2:4 model served in FP8 by vLLM reaches 27% higher prefill throughput than the dense FP8 model, with GSM8K accuracy on par.1

Sources (1)

• [1]Is 2:4 Sparsity Still Worth It? Semi-Structured Sparsity versus Quantization in LLM Inference
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 09:48 PM

  NVIDIA tensor cores have supported 2:4 semi-structured sparsity since Ampere, and the pattern composes with quantization.

  When the two are combined, the practical question is which step to take next: from a given precision, add 2:4 sparsity or drop to the next lower precision?

Extractive summary: sentences quoted from the sources.