Towards instance-dependent regret optimality in Episodic MDPs with Posterior Sampling

Key points

• While minimax-optimal algorithms are known, tractable approaches to instance-dependent optimality are still lacking.1
• Motivated by this gap, we introduce $π0$-PSRL, a variant of PSRL that uses posterior samples to decide when exploration is needed, while following a fixed reference policy $π0$ during exploration episodes.1
• We prove instance-dependent regret bounds for $π0$-PSRL, identifying the logarithmic exploration cost induced by $π0$ and taking a step toward matching asymptotic lower bounds for episodic RL.1
• Our proof techniques showcase a novel proof structure to derive problem-dependent regret bounds in episodic MDPs, and concentration results for Dirichlet random variables, that may be of independent interest.1

Sources (1)

• [1]Towards instance-dependent regret optimality in Episodic MDPs with Posterior Sampling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 08:51 AM

  We study regret minimization in finite-horizon episodic Markov Decision Processes (MDPs).

  While minimax-optimal algorithms are known, tractable approaches to instance-dependent optimality are still lacking.

Extractive summary: sentences quoted from the sources.