How Do DiTs Denoise? An Aggregator-Predictor View of Attention and MLPs

Key points

• How do diffusion transformers (DiTs) represent image content and learn rules for reusing it to generalize?1
• We first establish that, for Gaussian Markov random field (GMRF) priors, the global Bayes denoiser can be approximated using only a subset of context patches, with error decaying exponentially in context radius.1
• We further show that patchwise linear denoisers, often invoked to explain diffusion generalization, recover the empirical local denoiser with sufficiently informative aggregation.1
• DiTs trained on real images exhibit noise-dependent denoising contexts resembling those in the analytic GMRF setting, while attention mass exhibits a qualitatively corresponding broadening pattern.1

Sources (1)

• [1]How Do DiTs Denoise? An Aggregator-Predictor View of Attention and MLPs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 03:14 PM

  We develop an aggregator--predictor view of DiTs as input-adaptive local denoisers, in which context aggregation and patch prediction provide functional roles for interpreting attention and MLPs.

  How do diffusion transformers (DiTs) represent image content and learn rules for reusing it to generalize?

Extractive summary: sentences quoted from the sources.