Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding

Key points

• Lossless verification preserves the target model's output distribution and thus expected completion length, making this metric a meaningful efficiency proxy.1
• Relaxed (lossy) verification accepts tokens that lossless verification would reject, potentially altering that distribution.1
• We identify a length inflation phenomenon: relaxed verification lengthens completions, primarily by extending reasoning traces, largely offsetting apparent gains in accepted tokens per round.1
• We evaluate existing training-free relaxed verification rules on twelve target-drafter pairs against lossless verification on identical prompts, measuring completion length, verifier rounds, wall-clock time, and task accuracy.1

Sources (1)

• [1]Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 05:41 PM

  Speculative decoding typically characterizes speedup by draft tokens accepted per verification round.

  Lossless verification preserves the target model's output distribution and thus expected completion length, making this metric a meaningful efficiency proxy.

Extractive summary: sentences quoted from the sources.