Hierarchical Masked Diffusion Language Models

Key points

• Masked diffusion language models have emerged as an alternative to autoregressive models, enabling parallel token generation.1
• However, standard masked diffusion language models use a shared noise schedule across positions without explicitly encoding which positions should be resolved first.1
• It factors generation into two passes: the first predicts a hierarchy distribution over masked positions, the second predicts token distributions conditioned on it.1
• Our evaluation shows that for the task of language generation on LM1B, HMDLM reduces perplexity by more than 1.5 points over Anchored Diffusion Language Model (ADLM), a recently proposed strong diffusion baseline; on WikiHow, we see a drop in perplexity of up to 4.4 pts on ADLM and up to 7 pts against MDLM.1

Sources (1)

• [1]Hierarchical Masked Diffusion Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 06:06 AM

  To incorporate this notion of priority (or hierarchy) among token positions, we introduce the Hierarchical Masked Diffusion Language Model (HMDLM), which augments each token position with a hierarchy level available as part of the training data, and requires zero or minimal additional annotations.

  Masked diffusion language models have emerged as an alternative to autoregressive models, enabling parallel token generation.

Extractive summary: sentences quoted from the sources.