BERT Is Already a Denoiser: Cheaply Unlocking Iterative Generation in Masked Language Models

Key points

• We test whether this capability supports conditional generation without diffusion training.1
• Frozen BERT already exhibits substantial denoising capability, reconstructing 61.0%, 37.5%, and 24.4% of masked tokens at 15%, 45%, and 60% corruption, respectively, although generation deteriorates sharply as visible context disappears.1
• On open-ended conditional generation, full MLM adaptation reaches BLEU/ROUGE-L of 0.163/0.583 on QQP and 0.107/0.327 on Quasar-T, placing it in the same broad performance range as published BERT models trained through diffusion without explicit diffusion training.1
• Finally, applying the same generative interface to stronger masked representations substantially improves performance: RoBERTa-large reaches 44.2% on ARC-Easy and 33.8% on MMLU.1

Sources (1)

• [1]BERT Is Already a Denoiser: Cheaply Unlocking Iterative Generation in Masked Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:43 PM

  Masked language models (MLMs) such as BERT are typically used for representation learning despite their denoising pretraining.

  We test whether this capability supports conditional generation without diffusion training.

Extractive summary: sentences quoted from the sources.