Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability

Key points

• The lack of state observability may require optimal policies to depend on arbitrarily long observation-action histories, making planning and learning challenging.1
• We leverage recent advances in finite-memory approximations of POMDPs and represent finite history windows as states of a constrained MDP.1
• We then develop a primal--dual algorithm that provides the first non-asymptotic guarantees that jointly bound average reward suboptimality and constraint violation in the original CPOMDP.1
• Under a filter-stability assumption, we derive bounds relating optimality and feasibility in the finite-state model to those in the original CPOMDP, with approximation errors that decay geometrically with the window length.1

Sources (1)

• [1]Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 09:12 AM

  We study reinforcement learning in constrained partially observable Markov decision processes (CPOMDPs), where an agent maximizes reward subject to a cost constraint using only incomplete information about the underlying state.

  The lack of state observability may require optimal policies to depend on arbitrarily long observation-action histories, making planning and learning challenging.

Extractive summary: sentences quoted from the sources.