Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity

Key points

• Large Language Model inference over long contexts is increasingly bottlenecked by the cost of attending to a growing Key-Value (KV) cache.1
• Sparse attention reduces this cost by accessing only a fraction of the cache, but existing methods face a trade-off between deterministic retrieval, which captures high-scoring tokens but discards residual attention mass, and sampling, which improves coverage at the cost of higher variance.1
• Hybrid methods combine top-$k$ retrieval with uniform sampling, yet still exhibit a substantial gap from dense attention at aggressive sparsity.1
• With optimized CUDA kernels, SSA achieves up to $2.96\times$ decode speedup over FlashInfer at 1M context, demonstrating that its accuracy gains translate into practical long-context inference acceleration.1

Sources (1)

• [1]Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:02 PM

  We introduce Stratified Sparse Attention (SSA), a unified framework combining deterministic retrieval with score-guided sampling.

  Large Language Model inference over long contexts is increasingly bottlenecked by the cost of attending to a growing Key-Value (KV) cache.

Extractive summary: sentences quoted from the sources.