ResearchResearch paperReasoning & Planning · Reinforcement Learning · Training & Scaling2 sources · Oct 14, 2026

VASPO: Self-Distillation with Privileged Value Estimation

On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher.

Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally.
These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes.
The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios.
Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves.

[1]VASPO: Self-Distillation with Privileged Value Estimation
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 05:17 PM
On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher.
Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally.
[2]VASPO: Self-Distillation with Privileged Value Estimation
Hugging Face Daily Papers · Oct 14, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.

Before this