Bridged Self-Distillation for On-Policy Learning
To this end, we propose Bridged Self-Distillation (BSD), a unified framework for distilling from any teacher that provides reasoning traces.
ProofPaper ↗
Key points
- On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student.
- However, distillation from a stronger teacher model faces two mismatches: its solutions may be difficult for a smaller student to imitate, and its token-level probabilities may be incompatible with, or even unavailable to, the student.
- Existing methods handle these settings through different forms of supervision.
- The model to be distilled from, termed the external teacher, first provides a reasoning trace as privileged context, which can be generated in an offline fashion.
Sources (1)
- [1]Bridged Self-Distillation for On-Policy LearningarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 01:40 PM
To this end, we propose Bridged Self-Distillation (BSD), a unified framework for distilling from any teacher that provides reasoning traces.
On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026Selective Transfer through Sequence Distillation: From Benchmark Knowledge to Backdoor Behaviour
- Oct 19, 2026FLMNet: Self-Ensemble Distillation from Masks to Seeds for Single-Tower CTR Prediction
- Oct 19, 2026Functional Cross-layer Experts Routing for One-step Diffusion Distillation
- Oct 19, 2026ToolOPD: Credit-Aware On-Policy Distillation for Tool-Use Agents
- Oct 18, 2026ClinSDT: LLM-Encoded Clinical Semantic Guidance for Sepsis Treatment via Decision Transformer
- Oct 18, 2026Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation