Overtaught, Underlearned: Learner-Centered Distillation from Multi-Agent Discussion

Key points

• Full-graph distillation transfers such discussions by treating each correct solution as a separate training target, but what teachers provide need not match what the student learns.1
• This mismatch appears in two ways: several correct solutions may amount to the same lesson in the student's representation (over-teaching), while some training questions remain missed after learning (under-learning).1
• We quantify both with rates that are high on discussion graphs.1
• To address them, we introduce FAR (Fold--Attempt--Revisit), a learner-centered distillation procedure that organizes supervision around the student's own state.1

Sources (1)

• [1]Overtaught, Underlearned: Learner-Centered Distillation from Multi-Agent Discussion
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 10:27 AM

  Multi-agent discussion produces reasoning collaboratively, while deployment still relies on a smaller student that must answer alone.

  Full-graph distillation transfers such discussions by treating each correct solution as a separate training target, but what teachers provide need not match what the student learns.

Extractive summary: sentences quoted from the sources.