Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding

Key points

• D-PACE (Wu et al., 2026) trains such drafters with a loss that weights each position by its contribution to the expected accepted length.1
• Both its weights and its cross-entropy (CE) loss use only the draft's probability of the recorded target token, treating the target as a point mass.1
• We derive D-PAL (Dynamic Position-wise Acceptance Loss) from the exact expected accepted length, using this overlap as the acceptance signal.1
• We show that forward KL divergence and Rényi divergence of order 1/2 match or exceed LA, and that Rényi-1/2 provably bounds the true acceptance rate from above and below.1

Sources (1)

• [1]Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 03:20 PM

  In speculative decoding, a lightweight draft model proposes tokens that the target model verifies in parallel, speeding up generation without changing the target distribution.

  D-PACE (Wu et al., 2026) trains such drafters with a loss that weights each position by its contribution to the expected accepted length.

Extractive summary: sentences quoted from the sources.