Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?
On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision.
ProofPaper ↗
Key points
- We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories.
- Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections.
- We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability.
- Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.
Sources (1)
- [1]Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:11 PM
On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision.
We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Run fast, private agents on-device with LFM2.5-24B-A2B
- Oct 13, 2026GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning
- Oct 13, 2026Agent-Reflex: Teaching Language Agents to Act Reflectively
- Oct 13, 2026Cowork-Gym: Synthesizing Interactive Work Environments at Scale
- Oct 12, 2026JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models
- Oct 12, 2026A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes