RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies

Key points

• We formalize the design considerations for converting pretrained attention into a natively reduced-dimensional form that reduces both KV-cache storage and attention computation using a calibration set.1
• Specifically, we consider reductions that (1) apply a shared projection to queries and keys so that they remain in a common reduced feature space, (2) derive this projection jointly from query and key activations to better preserve their interaction, and (3) are compatible with the pretrained RoPE transformation.1
• Building on this characterization, we introduce RACE, a post-training method that selects rotary pairs using their joint query-key activation energy.1
• Across three model families and different compression levels, RACE outperforms structured pruning and remains competitive with reconstruction-based methods on language-modeling, zero-shot, and long-context tasks.1

Sources (1)

• [1]RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 03:34 PM

  Head-dimension compression can reduce the memory required by the key-value (KV) cache, but existing methods often reconstruct full-dimensional representations during inference, adding significant decoding FLOPs overhead.

  We formalize the design considerations for converting pretrained attention into a natively reduced-dimensional form that reduces both KV-cache storage and attention computation using a calibration set.

Extractive summary: sentences quoted from the sources.