A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus

Key points

• Under generic moment and smoothness conditions, the convergence analysis of the stochastic gradient method reduces to a calculus argument verifying a Lyapunov generator inequality for the SDE.1
• The method recovers established rates for classical stochastic gradient methods and provides a way to establish convergence rates for new optimization algorithms through calculus arguments applied to their associated SDEs.1

Sources (1)

• [1]A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:09 PM

  We develop a rate-transfer method that converts finite-time Lyapunov estimates for an associated stochastic differential equation (SDE) into finite-time convergence bounds for stochastic gradient algorithms.

  Under generic moment and smoothness conditions, the convergence analysis of the stochastic gradient method reduces to a calculus argument verifying a Lyapunov generator inequality for the SDE.

Extractive summary: sentences quoted from the sources.