Near Minimax Rate for Quantile Distributional Reinforcement Learning

Key points

• We investigate the statistical accuracy of estimating $\bmηm$ across quantile resolutions under Wasserstein distances.1
• Assuming access to a generative model, we construct a quantile fixed point estimator $\bmηm^{(n)}$ from an empirical Markov decision process and establish non-asymptotic error bounds under Wasserstein distances.1
• When reward densities vanish at the support endpoints, we show that continuity and endpoint monotonicity suffice to preserve the same resolution-independent rate for $1\le p\le2$.1
• To investigate the sharpness of these bounds, we characterize the exact local asymptotic minimax risk at each fixed resolution as the expected Wasserstein-coordinate norm of the efficient Gaussian limit and prove that our estimator attains it.1

Sources (1)

• [1]Near Minimax Rate for Quantile Distributional Reinforcement Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 11:35 AM

  In this paper, we study quantile distributional policy evaluation, where the return distribution under a given policy is approximated by the quantile Bellman fixed point $\bmη_m$.

  We investigate the statistical accuracy of estimating $\bmη_m$ across quantile resolutions under Wasserstein distances.

Extractive summary: sentences quoted from the sources.