A Nonlinear Time-Series Theory for Constant-step SGD with Momentum

Key points

• We study this coupled recursion as a nonlinear time series and develop a framework for its stationary behavior and distributional limits.1
• The joint parameter-momentum process forgets its initialization geometrically under a finite $p$th moment for $p > 1$, yielding a unique stationary causal solution.1
• Under a second-moment condition, this contraction leads to an invariance principle and quenched Gaussian limit for averaged SGDM, while a small-learning-rate analysis gives a Gaussian limit for the last iterate.1
• In a stochastic-curvature quadratic model, we further study how the rate of initialization forgetting relates to stationary variance and averaged fluctuations.1

Sources (1)

• [1]A Nonlinear Time-Series Theory for Constant-step SGD with Momentum
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 01:07 AM

  Constant-step stochastic gradient descent with momentum (SGDM) augments the parameter recursion with a momentum state that carries geometrically decaying memory of past stochastic gradients.

  We study this coupled recursion as a nonlinear time series and develop a framework for its stationary behavior and distributional limits.

Extractive summary: sentences quoted from the sources.