Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference

Key points

• Although token-level sparse attention reduces this burden by restricting computation to a small subset of relevant past tokens, its practical gains are often offset by repeatedly recomputing layer-wise sparse indices as the context grows.1
• A common remedy is to share sparse indices across layers by computing them only at designated anchor layers, but such static reuse ignores the layer-wise evolution of attention patterns, causing the reused indices to progressively deviate from the layer-specific optima and incur substantial attention loss.1
• By revisiting cross-layer sparsity from the perspective of attention dynamics, we find that inter-layer salient token shift is highly localized: although the exact top-$κ$ indices differ across layers, the core high-attention tokens remain stable, with changes concentrated near the expanded boundary of previously selected regions.1
• Motivated by this, we propose Prism Attention, a proposal-refined sparse attention framework.1

Sources (1)

• [1]Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 06:11 AM

  Long-context decoding in large language models (LLMs) is increasingly bottlenecked by the memory and computation required to attend over an ever-growing Key-Value (KV) cache at each autoregressive step.

  Although token-level sparse attention reduces this burden by restricting computation to a small subset of relevant past tokens, its practical gains are often offset by repeatedly recomputing layer-wise sparse indices as the context grows.

Extractive summary: sentences quoted from the sources.