DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

Key points

• On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use.1
• When divergence triggers an intervention during reasoning, the student prefix continues to receive standard OPD supervision.1
• The teacher then generates a recovery suffix, which the student learns through behavior cloning, using cross-entropy on the teacher-generated tokens.1
• We evaluate DTDD in three regimes: long-horizon reasoning, distillation from a base initialization without offline supervised fine-tuning (SFT), and multi-turn agents.1

Sources (1)

• [1]DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 11:26 PM

  We introduce Divergence-Triggered Dynamic Distillation (DTDD), an adaptive OPD framework that uses segment-level student-teacher divergence to trigger teacher takeover during training.

  On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use.

Extractive summary: sentences quoted from the sources.