From POMDP Theory to Deep RL with Particle Filters

Key points

• Model-based reinforcement learning in partially observable environments requires inferring hidden states, learning latent dynamics, and planning under the resulting belief.1
• Practical algorithms often solve one aspect of the above pipeline while relying on approximations for the others that are often poorly understood, while existing POMDP theory typically analyzes computationally inefficient theoretical algorithms far from modern deep RL practice.1
• We analyze each approximation and prove an end-to-end regret bound decomposing the error into model estimation error, particle filter error, Monte Carlo rollout error, and a structural bias term induced by the QMDP approximation controllable by the planning depth that we further analyze in various special cases.1
• We instantiate our algorithm in an MPC-based deep RL framework, demonstrating its practicality on a variety of environments.1

Sources (1)

• [1]From POMDP Theory to Deep RL with Particle Filters
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 03:41 AM

  To fill this gap, we propose a practical end-to-end algorithm that approximately solves the hidden state filtering problem via sequential Monte Carlo (SMC), updates a latent dynamics model online, and performs approximate online planning for $d$ steps before bootstrapping leaves with a state-value critic (a QMDP approximation).

  Model-based reinforcement learning in partially observable environments requires inferring hidden states, learning latent dynamics, and planning under the resulting belief.

Extractive summary: sentences quoted from the sources.