SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

Key points

• Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts.1
• Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding.1
• Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length.1
• On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3.1

Sources (1)

• [1]SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 07:41 AM

  Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding.

  Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts.

Extractive summary: sentences quoted from the sources.