Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization

Key points

• Self-distillation policy optimization provides dense token-level supervision for LLM post-training through feedback-conditioned self-teachers.1
• However, directly matching the self-teacher can disrupt successful behaviors on correct rollouts while providing insufficient correction on incorrect ones.1
• We analyze APSD through divergence decomposition, reverse-KL barycentric geometry, and gradient analysis, highlighting the importance of gradient flow through student-dependent targets.1
• Experiments under reinforcement learning with verifiable rewards and rich feedback show that APSD achieves strong overall performance against reinforcement-learning and self-distillation baselines and matches SDPO performance with 2.6x fewer training steps on average across Qwen3 model scales.1

Sources (1)

• [1]Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 08:03 AM

  We propose Asymmetric Policy Optimization via Self-Distillation (APSD), which constructs correctness-dependent targets by mixing student and self-teacher distributions: interpolation preserves successful behaviors on correct rollouts, while extrapolation beyond the self-teacher strengthens correction on incorrect ones.

  Self-distillation policy optimization provides dense token-level supervision for LLM post-training through feedback-conditioned self-teachers.

Extractive summary: sentences quoted from the sources.