Correcting or Continuing? What On-Policy Distillation Learns from Student Errors

Key points

• While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails.1
• We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol.1
• We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence.1
• These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.1

Sources (1)

• [1]Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 09:10 AM

  On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision.

  While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails.

Extractive summary: sentences quoted from the sources.