CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer

Key points

• Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers.1
• However, differences in vocabulary and segmentation make teacher and student next-token distributions difficult to align.1
• For one-to-one alignments, CSD uses explicit common-token pairs as the shared support.1
• Across four teacher-student settings comprising Llama-3.2-1B and Gemma3-1B-PT students with Qwen and Phi teachers, CSD consistently achieves the highest average performance among the evaluated cross-tokenizer methods, outperforming the strongest alternative by up to 1.8 points and surpassing both continued pretraining and same-tokenizer distillation.1

Sources (1)

• [1]CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 08:18 PM

  We introduce Common-Support Distillation (CSD), a unified logits-based framework that constructs a shared support between the teacher and student at each aligned input chunk and matches the induced distributions with a single objective.

  Cross-tokenizer logits distillation enables knowledge transfer across model families, allowing students to learn from stronger or more specialized teachers.

Extractive summary: sentences quoted from the sources.