E8 Lattice Key Quantization for Low-Bit KV-Cache Compression

Key points

• In this work, we demonstrate that a uniform low-bit k2v2 configuration (2-bit keys and 2-bit values) can preserve near-baseline quality by using vector quantization (VQ) based on the $E8$ lattice for KV keys.1
• We show that structured VQ exploits correlations across adjacent dimensions to achieve signal-to-quantization-noise ratios (SQNR) higher than those of independent scalar quantization at the same key payload rate.1
• Across five LLM families, our $E8$ k2v2 approach yields \textasciitilde$6\times$ KV compression, roughly 20% smaller than scalar k3v2 (3-bit keys and 2-bit values), while retaining near-baseline long-context perplexity.1
• These results indicate that $E8$ lattice VQ improves quality per byte while keeping throughput at parity, suggesting that structured key quantization is a useful primitive for KV-compression stacks.1

Sources (1)

• [1]E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 06:22 AM

  Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints.

  In this work, we demonstrate that a uniform low-bit k2v2 configuration (2-bit keys and 2-bit values) can preserve near-baseline quality by using vector quantization (VQ) based on the $E_8$ lattice for KV keys.

Extractive summary: sentences quoted from the sources.