Training-Free On-Policy Distillation

Key points

• On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher.1
• When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs.1
• We call this training-free on-policy distillation (\method).1
• What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision.1

Sources (1)

• [1]Training-Free On-Policy Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:15 PM

  We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone.

  On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher.

Extractive summary: sentences quoted from the sources.