Who Gets Unmasked? Sampler-Induced Bias in Diffusion Language Models

Key points

• Diffusion language models (dLLMs) generate by gradually unmasking a canvas, so their samplers decide not only which token to write but also when to commit it.1
• To separate the sampler from the model, we read the model's belief about a fork, such as a protagonist's gender, in one forward pass, and show that this belief matches exact sampling from the model.1
• Measured against it, peeking samplers on LLaDA and Dream amplify occupational gender stereotypes and collapse opinion distributions, and confidence-based ordering shifts gender-balanced occupations toward men, more so under faster parallel decoding.1
• We propose Fork-First Sampling, which decides the fork before anything else is written: it follows the belief or any target distribution, improves fluency, and avoids the stories that remark on the protagonist's gender, which prompt-based fixes produce, at the cost of one forward pass.1

Sources (1)

• [1]Who Gets Unmasked? Sampler-Induced Bias in Diffusion Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 06:02 PM

  We show that this second decision changes who appears in generated text, a sampler-induced bias on top of the model's own.

  Diffusion language models (dLLMs) generate by gradually unmasking a canvas, so their samplers decide not only which token to write but also when to commit it.

Extractive summary: sentences quoted from the sources.