Non-asymptotic Convergence of Average-reward Q-learning with Options
We study the finite-sample convergence of Q-learning algorithms for the average-reward formulation of Reinforcement Learning (RL) with options.
ProofPaper ↗
Key points
- While options allow temporarily extended and hierarchical actions, the average-reward formulation incorporates continual tasks beyond discounted and episodic RL.
- First, we prove that both inter- and intra-option Bellman operators satisfy multi-step span contractions under unichain assumptions.
- Then, we propose synchronous estimators of these multi-step option Bellman operators leading to two variants of option Q-learning.
- Finally, we extend the stochastic approximation framework to prove convergence rates of these two algorithms under the minimal assumptions used in the literature.
Sources (1)
- [1]Non-asymptotic Convergence of Average-reward Q-learning with OptionsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 03:26 PM
We study the finite-sample convergence of Q-learning algorithms for the average-reward formulation of Reinforcement Learning (RL) with options.
While options allow temporarily extended and hierarchical actions, the average-reward formulation incorporates continual tasks beyond discounted and episodic RL.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026PrismaCoder: Pushing the Boundaries of Open Multimodal Code Intelligence through Decomposed Rubric Synthesis
- Oct 14, 2026Player-1: Learning to Play from Synthetic Experience
- Oct 14, 2026The Design Space of Tri-Modal Masked Diffusion Models
- Oct 14, 2026More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
- Oct 13, 2026Run fast, private agents on-device with LFM2.5-24B-A2B
- Oct 13, 2026Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes