Posterior-Averaged Bellman Flow Matching for Distributional Critics

Key points

• Distributional critics model the full return distribution, providing richer signals for exploration and risk-sensitive control.1
• Recent work has shown that flow-based critics can model these distributions flexibly and provide effective supervision across flow time for temporal-difference learning.1
• However, unlike standard flow matching with fixed data endpoints, a flow critic is trained on Bellman targets recursively generated by a learned target critic under the current policy.1
• Uniform anchor selection makes this target an exact finite-pool Rao--Blackwellization of the single-endpoint target: for every \(K\), it preserves the population target, cannot increase conditional variance, and its conditional variance scales as \(O(K^{-1})\).1

Sources (1)

• [1]Posterior-Averaged Bellman Flow Matching for Distributional Critics
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 06:16 AM

  We propose Posterior-Averaged Bellman Flow Matching (PA-BFM), which samples an endpoint pool, infers the closed-form posterior over the latent anchor, and regresses to the posterior-mean velocity.

  Distributional critics model the full return distribution, providing richer signals for exploration and risk-sensitive control.

Extractive summary: sentences quoted from the sources.