JetSpec++: Foresight Parallel Drafting with Self Refinements

Key points

• Speculative decoding's effectiveness is governed by two quantities: the acceptance rate of drafted tokens and the cost of producing them.1
• JetSpec++ combines three components: intra-block self refinement, which trains the draft model to iteratively correct its own parallel predictions within a block of tokens; inter-block foresight drafting, which begins the next block before the current one is verified; and pipelined drafting and verification, which overlaps the two stages so drafting's marginal wall-clock cost approaches zero.1
• In comparison with state-of-the-art speculative decoding methods, JetSpec++ attains a wall-clock speedup of up to $7.45\times$ over autoregressive (AR) decoding and $1.54\times$ over DSpark, with up to 1,361 tokens per second for a single request on GB200, and an average acceptance length of 8.94 across seven benchmarks, while preserving the target model's output distribution.1

Sources (1)

• [1]JetSpec++: Foresight Parallel Drafting with Self Refinements
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 08:37 AM

  We show the two objectives can be co-optimized with algorithm-system co-design, and present JetSpec++, which retains a strong learned draft model while hiding its cost almost entirely.

  Speculative decoding's effectiveness is governed by two quantities: the acceptance rate of drafted tokens and the cost of producing them.

Extractive summary: sentences quoted from the sources.