SimOPD: Relational Knowledge Transfer for On-Policy Distillation
Inspired by contrastive learning's use of relationships between instances, we propose SimOPD to transfer teacher internal knowledge through response similarities.
ProofPaper ↗
Key points
- A key challenge in on-policy distillation (OPD) is extracting effective supervision from teacher knowledge.
- Supervision from token probabilities may underutilize internal knowledge, while direct hidden-state matching forces the student to shift to the teacher's feature space and complicates cross-architecture transfer.
- Specifically, SimOPD uses paired memory banks of student-generated responses to extract supervision from the teacher's internal representations as similarity distributions.
- The student then matches these distributions within its own representation space, retaining representational freedom and removing the need for cross-model projection.
Sources (1)
- [1]SimOPD: Relational Knowledge Transfer for On-Policy DistillationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 11:10 AM
Inspired by contrastive learning's use of relationships between instances, we propose SimOPD to transfer teacher internal knowledge through response similarities.
A key challenge in on-policy distillation (OPD) is extracting effective supervision from teacher knowledge.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation
- Oct 15, 2026Multi-Teacher On-Policy Distillation for Flow-Based Vision-Language-Action Models
- Oct 15, 2026Does On-Policy Distillation Need to Complete Every Rollout?
- Oct 15, 2026Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation
- Oct 15, 2026Condition-Contrastive Extrapolation for On-Policy Distillation of Language Models
- Oct 14, 2026Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models