Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets
On-policy learning is central to post-training: training on a model's own generations limits distribution shift and helps preserve existing capabilities.
ProofPaper ↗
Key points
- On hard problems, however, on-policy RL provides no reward-based learning signal when all attempts fail.
- On-policy distillation provides next-token supervision from a teacher, but on hard problems, if a rollout goes wrong early, subsequent supervision is conditioned on prefixes that already contain the mistake.
- Correct behavior can instead come from a gold or teacher solution, but such off-policy targets can differ substantially from the model's generations and, as we show, may erode existing capabilities.
- Theoretically, we prove that the student's learning regret grows with the edit size: it scales linearly with the horizon under minimal correction, as in DAgger, and quadratically under full-answer replacement, as in behavior cloning.
Sources (1)
- [1]Projected Distillation: On Learning to Solve Hard Problems from Close, Correct TargetsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 01:38 AM
On-policy learning is central to post-training: training on a model's own generations limits distribution shift and helps preserve existing capabilities.
On hard problems, however, on-policy RL provides no reward-based learning signal when all attempts fail.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
- Oct 19, 2026Bridged Self-Distillation for On-Policy Learning
- Oct 19, 2026Honeyed Words, Hidden Daggers: Benchmarking and Training LLMs to See Through Mock Politeness
- Oct 19, 2026ToolOPD: Credit-Aware On-Policy Distillation for Tool-Use Agents
- Oct 18, 2026TILDE: Teacher-in-the-Loop Distillation from Embodied Foundation Models
- Oct 18, 2026TIRE: Top-Down Formula Instantiation with Reliable Execution for Physics Problem Synthesis