Purify Before You Align: Improving Representation Alignment for Diffusion Models

Key points

• We introduce local appearance purification (LAP), which improves generation quality and makes encoders such as MAE more effective alignment targets by removing locally predictable content and aligning to the residual.1
• Through controlled interventions within a fixed encoder, we show that effective alignment depends on what information the target emphasizes, not simply how much image information it contains.1
• To understand why purification benefits some encoders more than others, we derive an exact constant-readout test that measures how well the target can be matched without using the diffusion model's hidden state.1
• LAP improves generation quality across five pretrained representations (MAE, AIMv2, CLIP, DINOv2, DINOv3), with the largest gain for MAE: on ImageNet-256 with SiT-B/2, it reduces FID from $53.1$ to $42.3$.1

Sources (1)

• [1]Purify Before You Align: Improving Representation Alignment for Diffusion Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 06:11 PM

  Representation alignment improves and accelerates diffusion training by encouraging intermediate features to match a pretrained visual encoder.

  We introduce local appearance purification (LAP), which improves generation quality and makes encoders such as MAE more effective alignment targets by removing locally predictable content and aligning to the residual.

Extractive summary: sentences quoted from the sources.