GRAFT: Restoring the Missing Sampling Interface in Continuous Diffusion Language Models

Key points

• We show that this same principle carries a hidden cost: a collapse of the lexical long tail.1
• We trace the difference to what continuous diffusion models lack: with no step that ever draws a discrete word, every step is instead driven by an average over the candidates still in play, and averaging a rare candidate together with the common word competing for its slot lands on the common word.1
• We propose GRAFT, a training-free method that grafts this missing choice back onto a frozen model: while the model is still undecided, GRAFT commits the rare words the average would otherwise discard, then regenerates the sequence from scratch with those commitments held in place.1
• Across three ELF scales and on LangFlow, an independent continuous dLLM, GRAFT restores the rare-word share of generated text from 0.42-0.77x the human rate to 0.82-1.10x, a relative gain of 34-109%, and moves the entire word-frequency profile closer to human on every model tested, by 2.7x on ELF-B and 12x on LangFlow.1

Sources (1)

• [1]GRAFT: Restoring the Missing Sampling Interface in Continuous Diffusion Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 03:15 AM

  Continuous diffusion language models generate a whole sequence in parallel, never drawing a single word at any intermediate step; only the final step turns embeddings into text.

  We show that this same principle carries a hidden cost: a collapse of the lexical long tail.

Extractive summary: sentences quoted from the sources.