Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

Key points

• Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation.1
• Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations and diffusion/flow spaces.1
• Drawing on the discrete DLM and continuous image diffusion literature on joint diffusion, we diffuse multiple modalities in parallel.1
• We propose a general setup that allows per-modality samplers and schedules to enhance the interplay between modalities.1

Sources (1)

• [1]Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 05:38 PM

  In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead.

  Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation.

Extractive summary: sentences quoted from the sources.