GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation

Key points

• On-policy distillation (OPD) provides token-level teacher feedback, yet common sampled-token updates assign credit locally and omit the effects of earlier actions on subsequent feedback.1
• We establish a theoretical basis for this approach by showing that, under ideal on-policy assumptions, broadcasting raw summed feedback to every response token yields an unbiased estimate of the negative gradient of the sequence-level reverse-KL objective.1
• To support effective learning in practice, \method{} discounts later feedback that may become less reliable under distribution shift and uses same-prompt rollout groups to calibrate teacher guidance.1
• Across five Qwen3 teacher--student settings and six mathematical reasoning benchmarks, \method{} outperforms vanilla OPD in macro Avg@K in every setting, including gains of 10.9 and 11.5 percentage points in two settings with matched thinking patterns.1

Sources (1)

• [1]GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 07:22 AM

  We introduce \method{} (Group-Relative On-Policy Distillation), which decouples feedback computation from credit assignment by aggregating token feedback into calibrated sequence-level credit.

  On-policy distillation (OPD) provides token-level teacher feedback, yet common sampled-token updates assign credit locally and omit the effects of earlier actions on subsequent feedback.

Extractive summary: sentences quoted from the sources.