Virtual Head Attention
We propose Virtual Head Attention (VHA), which simultaneously improves all three vertices through two lightweight linear operations: Q Premix applies a near-identity transformation to queries within each KV group to recover virtual head diversity from halved physical query heads, and Linear Postmix fuses inter-head features via a low-rank $I + AB^\top$ residual structure.
ProofPaper ↗
Key points
- The design of attention mechanisms in long-context large language models faces an "impossible triangle" among KV cache size, training and prefill FLOPs, and model quality.
- Since VHA introduces no nonlinear operations, Premix can be folded into the query projection and Postmix into the output projection at inference time, reducing the model to a standard GQA-2 architecture that directly reuses all existing GQA inference optimizations.
- With only two KV heads, the group size in large models readily reaches 64, fully exploiting Tensor Core parallelism during decoding.
- Across four scales with 4K-context 50B-token pretraining, VHA consistently surpasses the standard GQA baseline at every scale ($+$0.05% to $+$0.81%), while reducing KV cache by $4\times$ vs.\ GQA-8 at dense scales and $2\times$ vs.\ GQA-4 at the MoE scale, and lowering training and prefill FLOPs by 7.9% (1.7B, $S{=}4096$).
Sources (2)
- [1]Virtual Head AttentionarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:39 AM
We propose Virtual Head Attention (VHA), which simultaneously improves all three vertices through two lightweight linear operations: Q Premix applies a near-identity transformation to queries within each KV group to recover virtual head diversity from halved physical query heads, and Linear Postmix fuses inter-head features via a low-rank $I + AB^\top$ residual structure.
The design of attention mechanisms in long-context large language models faces an "impossible triangle" among KV cache size, training and prefill FLOPs, and model quality.
- [2]Virtual Head AttentionHugging Face Daily Papers · Oct 13, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs
- Oct 12, 2026Z.ai GLM-Legal Beats Claude Fable 5 and Kimi K3 on Harvey LAB
- Oct 12, 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards
- Oct 12, 2026Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
- Oct 11, 2026E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
- Oct 11, 2026History Replays Itself: Unleashing Long-Context Capability in Linear Attention
