Understanding Target Probability Reuse in Speculative Decoding
Speculative decoding uses a target model to verify a tree of draft tokens in parallel.
ProofPaper ↗
Key points
- We study whether the resulting next-token distributions can help select the next tree.
- For unrestricted target models, we show that guaranteed exact reuse requires a distribution computed at the next round's complete context.
- Across six targets, we find that reranking first-token branches with a separately computed distribution at the current context improves expected acceptance over draft ranking, especially with small tree budgets.
- Our findings show that making target probabilities available before tree allocation can improve the efficiency of speculative decoding.
Sources (1)
- [1]Understanding Target Probability Reuse in Speculative DecodingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:02 AM
Speculative decoding uses a target model to verify a tree of draft tokens in parallel.
We study whether the resulting next-token distributions can help select the next tree.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 21, 2026Target Regeneration Shrinks the Gains of Draft-Training Objectives in Speculative Decoding
- Oct 21, 2026Think Harder Where It Matters: Item-Level Adaptive Recursion for Sequential Recommendation
- Oct 20, 2026Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 20, 2026Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
- Oct 20, 2026RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation