On-Policy Transition Distillation
On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on student-generated rollouts.
ProofPaper ↗
Key points
- We investigate whether depthwise prediction transitions, the changes in predictions between layers, can provide additional task-relevant supervision.
- Our diagnostic study finds that stronger teacher--student agreement in intermediate transitions is associated with higher answer correctness within questions, with correct responses exhibiting lower mean transition KL across all measured intermediate layer pairs.
- These observations motivate On-Policy Transition Distillation (OPTD), which augments final-layer distribution matching by aligning teacher and student layer-to-layer log-probability increments on student-generated rollouts.
- In the three-domain setting, OPTD recovers $\approx 90%$ of the overall teacher--student capability gap, compared with $\approx 75%$ for OPD.
Sources (1)
- [1]On-Policy Transition DistillationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 11:37 AM
On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on student-generated rollouts.
We investigate whether depthwise prediction transitions, the changes in predictions between layers, can provide additional task-relevant supervision.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
- Oct 20, 2026Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds
- Oct 19, 2026NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation
- Oct 19, 2026Asynchronous On-Policy Self-Distillation
- Oct 19, 2026ToolOPD: Credit-Aware On-Policy Distillation for Tool-Use Agents
- Oct 19, 2026Ropedia W0: Scaling Loco-Manipulation World Models across Embodiments with Human Experience