Learnable Low-Rank Polynomial Sketch for Effective Linear Attention

Key points

• Among existing linear attention approaches, recent studies have shown that polynomial kernels are particularly effective, as they exhibit sparse and spiky behavior similar to softmax attention, which emphasizes large dot products while suppressing irrelevant interactions.1
• In this paper, we show that, from the perspective of polynomial kernel approximation, existing linear attention methods are still suboptimal.1
• We propose Learnable Low-Rank Polynomial Sketch (LLoPS), a principled and flexible framework for approximating polynomial kernels with linear attention.1
• Our method learns a low-rank polynomial sketch that provably achieves a strictly smaller approximation error than existing approaches.1

Sources (1)

• [1]Learnable Low-Rank Polynomial Sketch for Effective Linear Attention
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 06:30 PM

  Softmax attention in Transformers suffers from quadratic complexity in sequence length, making it impractical for long-context applications.

  Among existing linear attention approaches, recent studies have shown that polynomial kernels are particularly effective, as they exhibit sparse and spiky behavior similar to softmax attention, which emphasizes large dot products while suppressing irrelevant interactions.

Extractive summary: sentences quoted from the sources.