From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning

Key points

• Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction.1
• Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, leading to high computational costs and failing to effectively leverage suboptimal trajectories in datasets.1
• The model learns a shared conditional mean velocity field in this space, constructing two flow mappings: from Gaussian-Induced Decision Sources toward positive behavioral decision regions, and from suboptimal decisions toward high-value decision regions.1
• Furthermore, we propose hierarchical chunk-level advantage to guide the flow direction toward high-return regions, where the high-level decisions cover multiple low-level actions, leading to sequence length compression from $T$ to $K \approx T/H$ and significantly lower computational overhead.1

Sources (1)

• [1]From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 03:41 PM

  We propose MeanFlow2Better (MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space.

  Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction.

Extractive summary: sentences quoted from the sources.