ResearchResearch paperTraining & Scaling · Reinforcement Learning · Large Language Models1 source · Oct 19, 2026

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

On-policy self-distillation (OPSD) supervises student-generated trajectories with a reference-conditioned version of the same base model.

Gradient measurements on Qwen3-1.7B at initialization reveal that correct and incorrect responses have similar individual gradient magnitudes, but incorrect responses have approximately three times the coherent gradient share.
A signal--noise model fitted at a small group size tracks alignment at larger group sizes without refitting.
Motivated by these measurements, we introduce correctness-filtered OPSD (C-OPSD), which generates one response per problem and distills only a subset of incorrect responses.
Together, the measurements and training results connect outcome-conditioned supervision redundancy to more compute-efficient distillation.

[1]Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:55 AM
On-policy self-distillation (OPSD) supervises student-generated trajectories with a reference-conditioned version of the same base model.
Gradient measurements on Qwen3-1.7B at initialization reveal that correct and incorrect responses have similar individual gradient magnitudes, but incorrect responses have approximately three times the coherent gradient share.

Extractive summary: sentences quoted from the sources.

Before this