Outcome-Routed Distillation and Reinforcement Learning for Diffusion Language Models

Key points

• On-policy distillation provides dense supervision on the student's own samples, but its targets still come from the teacher.1
• Verifier reinforcement learning (RL) instead uses an outcome signal, but it provides no direct constraint on capabilities outside the verifier.1
• Combining the two is difficult for diffusion language models (dLLMs), because bidirectional denoising makes the policy likelihoods required by ratio-based RL unreliable.1
• A domain-conditional recipe retains a 16.37-point tool-calling gain; a second seed gives 18.23 points and reproduces both preservation results, but the math result is seed-dependent.1

Sources (1)

• [1]Outcome-Routed Distillation and Reinforcement Learning for Diffusion Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 10:44 PM

  We develop a likelihood-free pipeline for a mixture-of-experts dLLM.

  On-policy distillation provides dense supervision on the student's own samples, but its targets still come from the teacher.

Extractive summary: sentences quoted from the sources.