Noise-Space Reward Alignment for Few-Step Generative Models

Key points

• Few-step generative models produce high-quality samples in one or a few evaluations.1
• While one could apply approaches developed for iterative diffusion and flow-based models, such as direct RL fine-tuning or test-time scaling, we find that neither transfers well to the few-step regime: direct fine-tuning suffers from severe reward over-optimization, and test-time scaling sacrifices the efficiency advantage of few-step generation.1
• The trained policy transports Gaussian noise to a reward-tilted noise distribution, whose pushforward distribution through the frozen generator is exactly the reward-tilted output distribution.1
• To improve sample efficiency during training, we use ELBO-based likelihood estimation that requires only sampled endpoints rather than simulated trajectories.1

Sources (1)

• [1]Noise-Space Reward Alignment for Few-Step Generative Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 02:30 AM

  We propose Noise Policy Optimization (NPO), which explicitly targets the alignment problem for few-step generative models.

  Few-step generative models produce high-quality samples in one or a few evaluations.

Extractive summary: sentences quoted from the sources.