Trajectory-Triggered Sparse Collaborative Decoding

Key points

• Dense assistance methods such as speculative decoding repeatedly verify draft tokens with a target model, keeping target computation coupled to generation.1
• We show that local uncertainty is a weak trigger: entropy, negative log-likelihood, logit margin, top-$k$ mass, and velocity features give near-chance failure prediction, with AUC 0.50-0.53.1
• In contrast, aggregate trajectory-shape features over the draft confidence path reach AUC 0.86 using a lightweight logistic predictor trained only from draft correctness labels.1
• Used online, learned sparse collaborative decoding improves over the best open-loop schedule by +5.3 to +20.0 accuracy points across five GSM8K draft-target pairs.1

Sources (1)

• [1]Trajectory-Triggered Sparse Collaborative Decoding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 03:24 PM

  We study sparse collaborative decoding, where a draft model generates most tokens and a target model is invoked only for short corrective segments.

  Dense assistance methods such as speculative decoding repeatedly verify draft tokens with a target model, keeping target computation coupled to generation.

Extractive summary: sentences quoted from the sources.