Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers

Key points

• Block-sparse attention reduces this cost by computing only selected attention blocks.1
• In this work, we observe that sparsifying attention in different layers and at different denoising steps affects final-video fidelity to substantially different degrees.1
• Building on these findings, we propose Allocating Layer-step Latency by Output Tests (ALLOT), a training-free sparsity allocation method.1
• Extensive experiments across four additional published pipelines on Wan2.1, Wan2.2, and HunyuanVideo show that ALLOT outperforms or ties the default in all nine settings.1

Sources (2)

• [1]Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers
  Hugging Face Daily Papers · Oct 21, 12:00 AM

  Full 3D attention is costly in video Diffusion Transformers.

  Block-sparse attention reduces this cost by computing only selected attention blocks.

• [2]Sparsity Should Be ALLOTted by Output: Training-Free Allocation of Block-Sparse Attention for Video Diffusion Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 09:01 AM · same content

Extractive summary: sentences quoted from the sources.