When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding

Key points

• We establish a comparison protocol spanning training, inference, and evaluation, remeasuring released checkpoints and retraining six drafter recipes on shared target-generated data across ten benchmarks and two target models.1
• Alignment changes the picture: gaps that appear large in published comparisons shrink substantially under shared training and inference conditions.1
• This narrowing motivates two deeper questions: how is acceptance improved at different costs, and when do these gains improve throughput?1
• We find that acceptance can be improved through nearly free objective and context changes or more expensive token-to-token conditioning.1

Sources (1)

• [1]When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 03:52 AM

  The throughput gains of speculative decoding depend on the trade-off between acceptance length and execution latency.

  We establish a comparison protocol spanning training, inference, and evaluation, remeasuring released checkpoints and retraining six drafter recipes on shared target-generated data across ten benchmarks and two target models.

Extractive summary: sentences quoted from the sources.