A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

Key points

• Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic.1
• Building on the PhiBE-Q formulation (arXiv:2606.21925), we develop a model-free PhiBE policy iteration (PhiBE-PI) framework that uses a single state-action critic while preserving the continuous-time action-ranking scale.1
• We further develop an implementation under linear function approximation and establish a local end-to-end suboptimality guarantee for the learned policy.1
• Our analysis quantifies how errors from time discretization, function approximation, and finite-sample policy evaluation propagate through policy iteration.1

Sources (1)

• [1]A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 08:52 PM

  In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available.

  Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic.

Extractive summary: sentences quoted from the sources.