Next Token or Vector? Learning How Far to Decode

Key points

• Standard autoregressive decoding advances one token at a time, even when several upcoming tokens can be predicted together.1
• Gaussian prediction heads represent future spans as continuous latent vectors, and a shared decoder converts the selected vector into token predictions in parallel.1
• A learned router selects the prediction length from the model's own hidden states and predicted Gaussian parameters, adapting generation granularity to the current context with single-token generation as a fallback.1
• Experiments with Llama-3.1-8B-Instruct on Alpaca, GSM8K, and HumanEval show that router-selected positions yield higher draft acceptance than applying the corresponding prediction length everywhere.1

Sources (1)

• [1]Next Token or Vector? Learning How Far to Decode
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 01:00 AM

  We propose CADENCE (Context-Adaptive DEcoding), a generation method that learns both what to draft and how far to draft.

  Standard autoregressive decoding advances one token at a time, even when several upcoming tokens can be predicted together.

Extractive summary: sentences quoted from the sources.