Asynchronous On-Policy Self-Distillation

Key points

• Unlike on-policy distillation (OPD), which depends on a separate teacher model, on-policy self-distillation (OPSD) leverages a single language model acting simultaneously as the student and the teacher, with the teacher conditioned on privileged information.1
• Since the teacher and student share parameters, AsyncOPSD eliminates the multi-model coordination overhead inherent to OPD architectures in asynchronous training.1
• To mitigate this, we introduce a distillation objective based on staleness-aware policy control and token-level entropy weighting.1
• Across three model sizes and mathematical reasoning benchmarks, AsyncOPSD improves the training throughput by $1.96$--$2.25\times$ over synchronous OPSD while increasing average benchmark accuracy.1

Sources (1)

• [1]Asynchronous On-Policy Self-Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 06:38 AM

  To address this, we introduce Asynchronous On-Policy Self-Distillation (AsyncOPSD), a framework that decouples rollout generation from learner updates to improve training throughput.

  Unlike on-policy distillation (OPD), which depends on a separate teacher model, on-policy self-distillation (OPSD) leverages a single language model acting simultaneously as the student and the teacher, with the teacher conditioned on privileged information.

Extractive summary: sentences quoted from the sources.