ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction

Key points

• Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass.1
• The backbone's own final layer predicts every horizon, so the supervision lands on the parameters that generate the next token, and Dynamic Depth Sampling (DDS) keeps deeper horizons affordable by drawing the training depth with probabilities that decay along depth.1
• In pretraining, added depth converts into downstream accuracy where auxiliary modules saturate, concentrating on reasoning and code at half the MTP training overhead of training every depth.1
• At inference, the pretrained model is its own drafter: it drafts beyond the depth it was trained for, and keeps its draft-target alignment on the open-ended, high-entropy generation where post-hoc drafters lose theirs.1

Sources (2)

• [1]ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction
  Hugging Face Daily Papers · Oct 17, 12:00 AM

  We propose Efficient Causal Self-Recursion (ECHO), which removes both costs.

  Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass.

• [2]ECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 04:18 PM · same content

Extractive summary: sentences quoted from the sources.