Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers

Key points

• Our study suggests that dense attention in pretrained DiTs contains reusable layer/head-specific high-mass structure that can be identified from a small offline calibration set and reused across future generations.1
• This leads to Sink-Sparse Hybrid Attention, a train-free attention replacement that reuses static sink masks and supplements them with a lightweight pooled-KV branch, while leaving model weights and sampling schedules unchanged.1
• On Z-Image-Turbo, it attains 1.57$\times$ end-to-end speedup with competitive GenEval2 performance.1
• These results indicate that reusable importance-aware sparse support can provide an effective train-free acceleration path for both T2V and T2I DiTs.1

Sources (1)

• [1]Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 05:50 AM

  Diffusion Transformers (DiTs) have become a common backbone for high-quality text-to-image and text-to-video generation, but repeated dense self-attention remains a major inference bottleneck as token sequences grow and the same operator is invoked across many denoising steps.

  Our study suggests that dense attention in pretrained DiTs contains reusable layer/head-specific high-mass structure that can be identified from a small offline calibration set and reused across future generations.

Extractive summary: sentences quoted from the sources.