OPDArena: Understanding Learning and Forgetting in On-Policy Distillation

Key points

• We present OPDArena, a framework comparing supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and OPD across five pretrained models and diverse teacher sources.1
• Teacher choice strongly shapes retention: students of RLVR-trained teachers improve on the target task while retaining prior capabilities, whereas students of SFT-trained teachers incur greater capability loss.1
• Across post-training methods, out-of-domain (OOD) distribution change shows a more consistent positive association with forgetting than in-domain change.1
• Controlled teacher comparisons show that the task-specific preferences teachers acquire through post-training make student updates more concentrated in-domain.1

Sources (1)

• [1]OPDArena: Understanding Learning and Forgetting in On-Policy Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 01:51 AM

  On-policy distillation (OPD) is increasingly used for language model post-training, yet how teacher supervision shapes learning and forgetting remains unclear.

  We present OPDArena, a framework comparing supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and OPD across five pretrained models and diverse teacher sources.

Extractive summary: sentences quoted from the sources.