Large Diffusion Models Need More Than EMA

Key points

• We study whether EMA is uniquely suited to diffusion training, or whether stronger alternatives have been underexplored.1
• Our theory shows that averaging a constant-step trajectory has an inherent optimization limitation, whereas LR decay and Schedule-Free optimization admit vanishing stationarity guarantees under standard nonconvex assumptions.1
• Our analysis further shows that all three strategies substantially reduce parameter variability, yet stronger stabilization is not always better: EMA with a constant LR retains comparatively higher parameter variability, while aggressive decay combined with EMA achieves the lowest checkpoint variability but also weakens continued optimization.1
• Overall, our results suggest that large diffusion models need more than parameter averaging alone: effective optimization requires balancing variance reduction with continued optimization.1

Sources (1)

• [1]Large Diffusion Models Need More Than EMA
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 05:13 AM

  Large-scale diffusion models are commonly trained with a constant learning rate (LR) and exponential moving average (EMA), despite the strong performance of LR decay and Schedule-Free optimization in training large language models.

  We study whether EMA is uniquely suited to diffusion training, or whether stronger alternatives have been underexplored.

Extractive summary: sentences quoted from the sources.