FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

Key points

• Existing training-free approaches restrict the context window, rescale positions, or augment recent context with retrieved distant entries.1
• Drafters, however, read target hidden states that already encode contextual and positional information, and their lightweight architecture makes full-history attention affordable.1
• We propose FOLD, a training-free method that preserves exact local positions and treats all distant entries as far memory at a shared position.1
• Across six target-drafter pairs and diverse benchmarks of up to 256K tokens, FOLD predominantly delivers the highest speedup, with the best case more than doubling the speedup over autoregressive decoding, from 2.08$\times$ to 4.22$\times$.1

Sources (1)

• [1]FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 03:55 PM

  Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification.

  Existing training-free approaches restrict the context window, rescale positions, or augment recent context with retrieved distant entries.

Extractive summary: sentences quoted from the sources.