TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

Key points

• Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters.1
• To enable end-to-end learning despite the non-differentiable top-K operator for token pruning, we introduce token-gated attention, a mechanism inspired by Mixture-of-Experts that re-weights attention logits using token importance scores.1
• Across diverse tasks, TokenGate achieves 1.4-11.3times faster time-to-first token (TTFT) and 1.5-3.4times lower peak memory at no accuracy degradation for prefill in transformers.1
• TokenGate natively extends both standard transformers (e.g., Qwen3) and hybrid-attention models (e.g., Qwen3.5) with simple lightweight adaptation comparable to standard parameter-efficient fine-tuning.1

Sources (2)

• [1]TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
  Hugging Face Daily Papers · Oct 19, 12:00 AM

  We introduce TokenGate, a small, drop-in component that extends existing trained models to enable end-to-end learnable token pruning over long-context inputs for fast and memory-efficient prefill.

  Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters.

• [2]TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:15 PM · same content

Extractive summary: sentences quoted from the sources.