GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

Key points

• In this paper, we ask whether the recurrent memory already knows what the attention memory will need.1
• We first show that which GDN heads of Qwen3.8-27B keep long memories is fixed by the weights rather than by the document.1
• Next, we define the future utility of a KV entry by how much removing it changes the attention output over the model's own continuations, and find that the write gate, decay and write magnitude that GDN layers emit during prefill anticipate this utility before any query is seen.1
• Because the relation is spread across many GDN heads and layers, and a linear map recovers its trend but less of its high-utility tail than a nonlinear one, we propose GateSight, which learns it with lightweight per-head multilayer perceptrons that read these signals and no attention score.1

Sources (1)

• [1]GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 09:19 AM

  Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV) cache grows with the context and dominates serving memory.

  In this paper, we ask whether the recurrent memory already knows what the attention memory will need.

Extractive summary: sentences quoted from the sources.