How GLM5.3 Sparse Attention Affects HBM Memory Usage

Key points

• How does sparse attention affect the TAM of memory, including HBM and NAND?1
• Sparse attention selects top-k most relevant tokens to attend to, reducing the memory consumption and bandwidth requirements during the core Scaled Dot-Product Attention (SDPA) operation.1
• To reduce the cache miss latency, HiSparse overlaps KV cache loading of layer N with the execution of layer N-1 (layer-wise overlapping), introduced in HiSparse’s prior work HiCache.1
• In addition, HiSparse shows that system optimizations can overcome sparse attention’s memory capacity limitations, so sparse attention memory profile alone cannot sufficiently portray the full picture of system KV cache efficiency.1

Sources (1)

• [1]How GLM5.3 Sparse Attention Affects HBM Memory Usage
  SemiAnalysis · Sep 28, 07:26 PM

  How Sparse Attention Affects DRAM/NAND Memory

  How does sparse attention affect the TAM of memory, including HBM and NAND?

Extractive summary: sentences quoted from the sources.