Distribution-Based Regularization for Reward Fine-Tuning Diffusion and Flow Models

Key points

• Existing methods attempt to control this drift with reverse-KL regularization, implemented as an $\ell2$ penalty on the velocity field; however, this functions by indirectly constraining sampling dynamics rather than directly acting on the generated distribution.1
• To address this, we propose DFR (Distributional Functional Rewards), which augments the task reward with a distribution-dependent, per-sample reward that acts explicitly on the generated distribution.1
• Formally, we generalize the KL-regularized objective from expected rewards, which are linear in the model's distribution, to convex functionals of it.1
• Across various text-to-image fine-tuning tasks, DFR matches the peak reward of KL-tuned baselines while increasing diversity by up to 29% at equal or lower drift.1

Sources (1)

• [1]Distribution-Based Regularization for Reward Fine-Tuning Diffusion and Flow Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 10:36 PM

  Reward fine-tuning is an effective way to adapt pretrained diffusion and flow-matching models to downstream objectives, but aggressive optimization often leads to mode collapse and distributional drift.

  Existing methods attempt to control this drift with reverse-KL regularization, implemented as an $\ell_2$ penalty on the velocity field; however, this functions by indirectly constraining sampling dynamics rather than directly acting on the generated distribution.

Extractive summary: sentences quoted from the sources.