Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation

Key points

• Building on this insight, we theoretically explore what makes a teacher and a student compatible for OPD, and leverage our findings to derive several empirical heuristics for teacher selection.1
• In particular, our framework captures several desiderata for OPD: compatible teacher-student reasoning patterns, teacher competence, and induced student diversity.1
• Theoretically, we categorize several types of teacher routing schemes, and we prove that under different scenarios, different routing schemes are likely to be beneficial for optimization and student diversity.1
• We back up our findings with extensive experimental results in domains such as math, coding, kernel generation, and visual reasoning, where we find upwards of 10 percentage points of improvement on pass@$k$ when compared to vanilla OPD and the teacher experts on certain tasks, indicating both capability improvement and diversity over high-quality responses.1

Sources (1)

• [1]Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 04:26 AM

  On-policy distillation (OPD) has recently been introduced as a promising paradigm for continual learning, where a student policy can be supervised at the token-level.

  Building on this insight, we theoretically explore what makes a teacher and a student compatible for OPD, and leverage our findings to derive several empirical heuristics for teacher selection.

Extractive summary: sentences quoted from the sources.