Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation

Key points

• While long chain-of-thought (CoT) distillation significantly improves the reasoning abilities of language models, its underlying mechanism remains poorly understood.1
• Current pipelines generally rely on rejection sampling to gather reasoning traces across diverse prompts, assuming that factual correctness and broad problem coverage are indispensable.1
• Surprisingly, we find that training exclusively on flawed reasoning trajectories or a restricted prompt set yields nearly comparable performance.1
• We reveal a critical interplay: while the recent probability-based loss excels at learning simple patterns, the standard cross-entropy loss is essential for scaling to complex CoTs.1

Sources (1)

• [1]Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 02:29 AM

  Driven by this counterintuitive finding, we propose the Behavioral Hypothesis: the essence of Long-CoT distillation is behavioral alignment with structured reasoning patterns, rather than factual knowledge memorization.

  While long chain-of-thought (CoT) distillation significantly improves the reasoning abilities of language models, its underlying mechanism remains poorly understood.

Extractive summary: sentences quoted from the sources.