RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding

Key points

• Speculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model.1
• Conventional training relies on soft-label distillation, which forces the draft model to mimic the target's full output distribution, including high-entropy regions that degrade acceptance rates and increase early rejection risk.1
• RCSD leverages naturally occurring accepted and rejected sequences during speculative inference to construct implicit preference pairs, with masked loss to avoid penalizing target-aligned tokens.1
• Evaluated across multiple benchmarks, RCSD improves average acceptance length by 35% over HASS and achieves 1.35x faster end-to-end inference than HASS on practical workloads, without compromising generation quality.1

Sources (1)

• [1]RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 10:16 AM

  We propose RCSD (Reinforcement and Continual Learning for Speculative Decoding), a training framework that aligns draft model optimization with the system-level objective of maximizing acceptance length.

  Speculative decoding accelerates large language model inference by employing a compact draft model to propose candidate tokens that are verified in parallel by a target model.

Extractive summary: sentences quoted from the sources.