Closing the Horizon Gap in Policy Optimization for Adversarial MDPs

Key points

• Policy optimization updates the policy locally at each state and avoids optimization over the occupancy-measure polytope, but its existing regret bounds are larger by a factor of the horizon $H$ than those of occupancy-measure-based algorithms.1
• We close this gap by using regularized $Q$-functions, which allow us to control the stability of the local updates jointly over all state-action pairs rather than separately at each state.1
• The resulting algorithm attains high-probability regret bounds of $\widetilde O(\sqrt{HS(H+A)T})$ for known transitions and $\widetilde O(HS\sqrt{AT})$ for unknown transitions, where $S$ is the number of states, $A$ the number of actions, and $T$ the number of episodes.1
• We further extend the algorithm to adversarial linear-mixture MDPs and obtain the same improvement in the horizon dependence.1

Sources (1)

• [1]Closing the Horizon Gap in Policy Optimization for Adversarial MDPs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:26 PM

  We consider policy optimization for online episodic tabular Markov decision processes (MDPs) with adversarial losses and bandit feedback.

  Policy optimization updates the policy locally at each state and avoids optimization over the occupancy-measure polytope, but its existing regret bounds are larger by a factor of the horizon $H$ than those of occupancy-measure-based algorithms.

Extractive summary: sentences quoted from the sources.