DLite: History-free Speculative Decoding for Agentic AI

Key points

• Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model.1
• However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions.1
• DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context, transforming an increasingly complex modeling problem into local simple prediction problem.1
• Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2-6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%.1

Sources (2)

• [1]DLite: History-free Speculative Decoding for Agentic AI
  Hugging Face Daily Papers · Oct 16, 12:00 AM

  In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI.

  Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model.

• [2]DLite: History-free Speculative Decoding for Agentic AI
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 05:18 PM · same content

Extractive summary: sentences quoted from the sources.