A Unified Bellman Operator for Safety-Critical Reinforcement Learning

Key points

• Existing safe reinforcement learning paradigms typically force a trade-off: they either require a priori knowledge to provide strict safety guarantees (e.g., safety filters), or they enable joint learning but only satisfy safety constraints on average.1
• In this work, we propose a novel Bellman operator that unifies performance and safety objectives into a joint value function.1
• We show that temporal difference learning with the joint Bellman operator converges under a two-timescale stochastic approximation framework.1
• Empirical evaluations on continuous control tasks with neural approximations demonstrate stable convergence with near-zero safety violations at test time.1

Sources (1)

• [1]A Unified Bellman Operator for Safety-Critical Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:53 PM

  Reinforcement learning in safety-critical domains requires maximizing task performance while strictly adhering to safety constraints.

  Existing safe reinforcement learning paradigms typically force a trade-off: they either require a priori knowledge to provide strict safety guarantees (e.g., safety filters), or they enable joint learning but only satisfy safety constraints on average.

Extractive summary: sentences quoted from the sources.