Teach Me After You Learn It: Lookahead Self-Distillation for RLVR
On-policy self-distillation (OPSD) has emerged as a popular training paradigm for LLMs, providing dense, token-level supervision by aligning a model's own distribution with the distribution induced by privileged information.
ProofPaper ↗
Key points
- We propose Lookahead Self-Distillation (LEAD), which introduces inter-instance distillation by allowing the teacher to first learn from other on-policy rollouts before teaching the student.
- LEAD constructs lookahead teachers through virtual RL updates on disjoint subsets of on-policy rollouts, and uses each teacher to provide cross-group token-level supervision for the other subset.
- To reconcile the complementary yet potentially conflicting signals from OPSD and RLVR, we introduce harmonious modulation, which adaptively modulates the GRPO advantage based on the agreement between the two signals.
- Extensive experiments on scientific reasoning and interactive-agent benchmarks show that LEAD achieves the best in-domain average performance across model scales while maintaining out-of-domain generalization comparable to GRPO.
Sources (1)
- [1]Teach Me After You Learn It: Lookahead Self-Distillation for RLVRarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 12:22 PM
On-policy self-distillation (OPSD) has emerged as a popular training paradigm for LLMs, providing dense, token-level supervision by aligning a model's own distribution with the distribution induced by privileged information.
We propose Lookahead Self-Distillation (LEAD), which introduces inter-instance distillation by allowing the teacher to first learn from other on-policy rollouts before teaching the student.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026LoOPSD: On-Policy Self-Distillation with Looped Teachers
- Oct 13, 2026RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning
- Oct 12, 2026Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planning
- Oct 12, 2026Self-Distillation with Verbal Regularization
- Oct 12, 2026Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards
- Oct 12, 2026Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR