Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation

Key points

• On-policy distillation (OPD) trains agents on their own trajectories using supervision from a stronger teacher.1
• We term this phenomenon distillation overshoot.1
• Through decision- and task-level analyses, we trace overshoot to two competing effects of OPD: repairing weak student decisions and eroding successful student choices that the teacher disfavors.1
• Motivated by this finding, we propose Success-Preserving On-Policy Distillation (SPOD), which selectively masks conflicting teacher supervision on successful student choices and introduces a soft probability floor to counteract their erosion, while retaining teacher guidance elsewhere.1

Sources (1)

• [1]Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 05:44 AM

  We uncover a counterintuitive phenomenon in multi-turn agentic tasks: a smaller, initially weaker student can substantially outperform its teacher during training, only to regress as distillation continues.

  On-policy distillation (OPD) trains agents on their own trajectories using supervision from a stronger teacher.

Extractive summary: sentences quoted from the sources.