Sample-Efficient Reinforcement Learning with General Utilities via Neural Dual Critics

Key points

• We develop a natural actor-critic method for concave RLGU objectives admitting an exact variational dual representation, with explicit constructions for a broad class of separable occupancy utilities.1
• Rather than estimating the occupancy measure and then forming the policy-gradient pseudo-reward, our method learns this pseudo-reward directly as a function of the state-action pair using a neural critic.1
• Our analysis separates conditional actor bias from estimator second moments and accounts explicitly for dual optimization, critic approximation, and neural linearization errors.1
• Under policy-transfer, dual-stability, and local critic-regularity conditions, the method achieves $\widetilde{O}(T^{-1/2})$ global convergence above policy, critic-class, and finite-width approximation errors using $\widetilde{O}(T)$ expected environment transitions and reference samples.1

Sources (1)

• [1]Sample-Efficient Reinforcement Learning with General Utilities via Neural Dual Critics
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 09:35 PM

  Reinforcement learning with general utilities optimizes nonlinear functionals of policy-induced occupancy measures.

  We develop a natural actor-critic method for concave RLGU objectives admitting an exact variational dual representation, with explicit constructions for a broad class of separable occupancy utilities.

Extractive summary: sentences quoted from the sources.