Preserving Updates in Low-Bit Recurrent State Quantization

Key points

• However, storing these states in high precision incurs substantial memory and bandwidth costs at large batch sizes, motivating low-bit state quantization.1
• Unlike weights and key--value caches, a recurrent state is requantized after every update, so its quantization errors accumulate across steps.1
• Before rounding, we propose Low-Rank Bypass (LRB), which accumulates small updates in a compact low-rank buffer and periodically merges its sufficiently large elements into the state.1
• Therefore, we propose Bounded Directional Rounding (BDR), which rounds reversed elements toward their updates without additional storage.1

Sources (1)

• [1]Preserving Updates in Low-Bit Recurrent State Quantization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 03:46 PM

  Linear attention and recurrent models compress past tokens into a fixed-size state that is updated at every step.

  However, storing these states in high precision incurs substantial memory and bandwidth costs at large batch sizes, motivating low-bit state quantization.

Extractive summary: sentences quoted from the sources.