Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

Key points

• However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training.1
• We investigate whether agreement between early and late teacher policy shifts can guide selective transfer.1
• We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes.1
• Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning.1

Sources (2)

• [1]Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 09:47 AM

  Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories.

  However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training.

• [2]Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
  Hugging Face Daily Papers · Oct 15, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.