PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States

Key points

• Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time.1
• Storing the state at lower precision therefore lets more sequences share an accelerator.1
• Additionally, we find that the large entries of the gated-delta state gather in columns, because some channels of the update written at each token remain large across many tokens.1
• On four hybrid models from 4B to 35B parameters, PACE with mixed four- and eight-bit heads is competitive with eight-bit DAMP on the three larger models at about two thirds of its memory, PACE with four-bit heads throughout scores higher than Q-Mamba at equal memory, and rounding every few tokens preserves the decoding speedup of the buffer.1

Sources (2)

• [1]PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States
  Hugging Face Daily Papers · Oct 21, 12:00 AM

  Existing state quantizers round the state after every generated token and report that four bits lose accuracy unless the bit width is raised or the model is fine-tuned.\par We introduce PACE, a post-hoc four-bit state quantizer.

  Linear-attention language models keep their context in a recurrent state of fixed size, where an inference system stores one copy of this state for every sequence it decodes at the same time.

• [2]PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 10:31 AM · same content

Extractive summary: sentences quoted from the sources.