Rethinking the design space of Vanilla Pixel Diffusion Transformers

Key points

• JiT shows that a vanilla diffusion transformer can generate directly in pixel space by replacing the commonly used velocity prediction ($\boldsymbol{u}$-prediction) with clean image prediction ($\boldsymbol{x0}$-prediction).1
• In this work, we revisit this choice and introduce MFlow, an adaptive parameterization that learns to interpolate between $\boldsymbol{x0}$- and $\boldsymbol{u}$-prediction across principal directions of the data, learned end-to-end with diffusion training.1
• In addition, we find that vanilla pixel diffusion transformers, irrespective of parameterization, retain a persistent high-frequency deficit relative to real images.1
• To address this, we introduce ConvRefiner, a lightweight convolutional module trained post-hoc with an adversarial loss while keeping the diffusion backbone frozen.1

Sources (1)

• [1]Rethinking the design space of Vanilla Pixel Diffusion Transformers
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 05:59 PM

  Pixel-space diffusion models have recently re-emerged as a compelling alternative to latent diffusion, avoiding the lossy latent bottleneck and separately trained autoencoder.

  JiT shows that a vanilla diffusion transformer can generate directly in pixel space by replacing the commonly used velocity prediction ($\boldsymbol{u}$-prediction) with clean image prediction ($\boldsymbol{x_0}$-prediction).

Extractive summary: sentences quoted from the sources.