Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token.
ProofPaper ↗
Key points
- A natural explanation of where it goes wrong is compounding error: the student's mistakes carry its trajectories beyond the region where the teacher is reliable, so recent methods keep training inside that region, through episode-length curricula, teacher-guided turns, prefix replay, or teacher intervention where the two disagree.
- We show this explanation is incomplete.
- We attribute this to the myopia of next-token scoring.
- The teacher rates each token without seeing where the episode ends, so its scores let through the safe, delaying moves that outcome-based RL would punish with zero reward, a penalty agentic OPD lacks.
Sources (1)
- [1]Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy DistillationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 05:40 PM
Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token.
A natural explanation of where it goes wrong is compounding error: the student's mistakes carry its trajectories beyond the region where the teacher is reliable, so recent methods keep training inside that region, through episode-length curricula, teacher-guided turns, prefix replay, or teacher intervention where the two disagree.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Building Telecom Language Models with RSOD: Domain Data, Specialist Post-Training, Memory-Conscious Distillation, and Networking Evaluation
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 20, 2026Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
- Oct 19, 2026Bridged Self-Distillation for On-Policy Learning
- Oct 19, 2026ToolOPD: Credit-Aware On-Policy Distillation for Tool-Use Agents
- Oct 19, 2026Cross-Pollination Retrieval: Finding Transferable Ideas Across Fields in Scientific Literature