Looped Uniform Diffusion Language Models

Key points

• During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context.1
• Recently, selective looping, which recursively applies a few early Transformer layers within each denoising step, has been shown to improve training efficiency in MDLMs under an iso-FLOP budget.1
• We find that looping significantly improves training efficiency: looped UDLMs reach the validation NLL of a non-looped UDLM trained on about 0.5 trillion tokens with up to 2.8x fewer training FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu.1
• Finally, we study different loop-count distributions and find that sampling smaller loop counts more frequently during training improves inference efficiency: on downstream tasks, this strategy uses 12% fewer loops than uniform loop-count sampling without degrading accuracy.1

Sources (2)

• [1]Looped Uniform Diffusion Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:24 PM

  Uniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised.

  During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context.

• [2]Looped Uniform Diffusion Language Models
  Hugging Face Daily Papers · Oct 21, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.