Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search

Key points

• Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention.1
• Sparse attention mitigates this by skipping unnecessary computation in the attention matrix, but existing methods apply a fixed sparsity ratio throughout denoising and struggle to accurately determine which computations to skip.1
• To address these limitations, we analyze attention dynamics through attention entropy and mean attention distance, and find that attention becomes increasingly concentrated across denoising steps.1
• GSRS adopts Top-p selection for adaptive sparsity control, supported by a precise evaluation scheme that computes exact attention weights within the FlashAttention computation at negligible overhead.1

Sources (1)

• [1]Sparse Attention for Video Generation Acceleration via Growing Sparsity and Reduced Search
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 02:37 AM

  Based on these findings, we propose Growing Sparsity and Reduced Search (GSRS), a training-free framework that retains more computation early and progressively reduces it later, while restricting subsequent searches to an early candidate set to reduce search overhead.

  Diffusion Transformers (DiTs) dominate video generation but suffer from high inference cost due to the quadratic complexity of self-attention.

Extractive summary: sentences quoted from the sources.