Deep Pontryagin Learning for Continuous-Time Stochastic Control with Jumps

Key points

• In contrast to direct policy optimization, which repeatedly differentiates the objective through the entire time-discretized dynamics, our method separates adjoint learning from policy improvement by explicitly incorporating state-sensitivity information from the Pontryagin stochastic maximum principle.1
• For a fixed Markov control, we first compute pathwise adjoint labels by backpropagating the reward through time along simulated state trajectories, and project these labels onto the Markov state through a supervised regression step.1
• The control is then updated by enforcing the pointwise Hamiltonian stationarity condition.1
• We prove that the resulting projected adjoint process satisfies a discrete-time Pontryagin principle and converges, as the time step tends to zero, to the solution of the corresponding continuous-time adjoint backward stochastic differential equation.1

Sources (1)

• [1]Deep Pontryagin Learning for Continuous-Time Stochastic Control with Jumps
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 04:49 PM

  We develop a deep Pontryagin learning framework for high-dimensional continuous-time stochastic control problems that naturally accommodates jump-diffusions.

  In contrast to direct policy optimization, which repeatedly differentiates the objective through the entire time-discretized dynamics, our method separates adjoint learning from policy improvement by explicitly incorporating state-sensitivity information from the Pontryagin stochastic maximum principle.

Extractive summary: sentences quoted from the sources.