DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss

Key points

• Knowledge distillation (KD) frameworks place teacher and student on the same GPUs, so per-GPU memory limits the teachers that fit.1
• Closed-form memory and traffic models locate the memory limits of each layout.1
• On the Qwen3.5-35B-A3B to 9B pair, the disaggregated layout matches the LM loss of the colocated run with the same loss and is faster than that run; our fused loss also lets a colocated run train on as few nodes at faster speed, while the split is memory efficient when the student spans several nodes.1
• On the larger pairs, colocated runs with a fused loss still run OOM where the split trains, because teacher weights and student states fill the budget.1

Sources (2)

• [1]DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 03:17 AM

  DisaggKD separates them across nodes: teacher replicas stream hidden states to the student, a KD loss over row windows and vocabulary shards equals the full-vocabulary loss in exact arithmetic without forming the full-vocabulary logits, and a prefetch ring paces producer and trainer.

  Knowledge distillation (KD) frameworks place teacher and student on the same GPUs, so per-GPU memory limits the teachers that fit.

• [2]DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss
  Hugging Face Daily Papers · Oct 13, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.