Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning

Key points

• Generative policies offer expressive action distributions for online reinforcement learning (RL), but many diffusion- and flow-based policies require costly iterative sampling.1
• To incorporate entropy without a tractable policy density, we build on an energy-based view of Drifting and derive critic-guided Drift and entropy-induced Spread from the negative MaxEnt objective.1
• Our method learns these updates sequentially in the same actor, using action regression for value improvement and distribution matching to capture the effect of entropy.1
• Under exact fitting with a fixed critic, we prove convergence to the Boltzmann target in the small-step limit followed by the long-time limit.1

Sources (1)

• [1]Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 02:40 PM

  We propose DriftSpread, a one-step generative policy based on Drifting for online maximum-entropy (MaxEnt) RL.

  Generative policies offer expressive action distributions for online reinforcement learning (RL), but many diffusion- and flow-based policies require costly iterative sampling.

Extractive summary: sentences quoted from the sources.