Uniform Discrete Diffusion Models are Minimax Optimal for Estimating Distributions with Small Effective Support Size

Key points

• We address this gap for uniform discrete diffusion, one of the two dominant discrete diffusion paradigms alongside masking diffusion, by deriving statistical guarantees governed by the effective support size $sn(P0)$, a sample-size-dependent measure of distributional complexity.1
• Given $n$ independent and identically distributed (i.i.d.) samples from an unknown data distribution $P0$ on $[K]^d$, we show that, with appropriate choices of network size and hyperparameters, the expected total variation (TV) loss scales as $O(\sqrt{sn(P0)/n})$, while the expected Kullback--Leibler (KL) divergence is bounded by $O(\frac{1}{n}sn(P0)\log(eK^d/sn(P0))\log n)$.1
• Furthermore, we show that the TV rate is minimax optimal and that the KL rate is minimax optimal up to a factor of $\log n$.1
• Together, these upper and lower bounds show that uniform discrete diffusion successfully avoids the curse of dimensionality for distributions with small effective support size: the TV error rate depends on the ambient state-space size only through $sn(P0)$, while the corresponding KL rate incurs only an additional logarithmic dependence on the ambient state-space size.1

Sources (1)

• [1]Uniform Discrete Diffusion Models are Minimax Optimal for Estimating Distributions with Small Effective Support Size
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 6, 02:50 AM

  Discrete diffusion models have emerged as a practically successful framework for generative modeling on discrete product spaces, yet their statistical generalization properties remain poorly understood.

  We address this gap for uniform discrete diffusion, one of the two dominant discrete diffusion paradigms alongside masking diffusion, by deriving statistical guarantees governed by the effective support size $s_n(P_0)$, a sample-size-dependent measure of distributional complexity.

Extractive summary: sentences quoted from the sources.