Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing

Key points

• Distillation of reasoning chains from Large Language Models (LLMs) into their smaller siblings is a standard training approach.1
• Frequently, researchers do not have full access to a teacher model and its internal representations, which limits the resulting performance.1
• Specifically, for three open models (3-4B), at each epoch, we sample reasoning chains weighted by entropy difference with an auxiliary medium model (70-72B) and, next, supplement thinking traces with single-token ground truth answers.1
• Additionally, we present a comparative analysis of methods to generate and normalize distillation traces.1

Sources (1)

• [1]Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 08:45 PM

  We propose a recursive offline distillation method combining two components: entropy-based sampling for selective distillation and answer mixing.

  Distillation of reasoning chains from Large Language Models (LLMs) into their smaller siblings is a standard training approach.

Extractive summary: sentences quoted from the sources.