Physics-Inspired Optimizers beyond the Edge of Stability: Period-Two Orbits and Delayed Instability

Key points

• In deep learning the learning rate is usually large, so training is a discrete dynamical system: in many full-batch experiments the sharpness grows to the optimizer's stability limit, the edge of stability (EoS), and training continues there.1
• Physics-inspired optimizers change this motion; Velocity-Regularized Adam (VRAdam), for example, shrinks the step at high speed.1
• VRAdam's damper fails above twice the critical learning rate, where its kinetic energy is not convex; the Legendre dual of VRAdam's own Lagrangian is convex and passes at every learning rate. (ii) Because the sharpness rises gradually, the loss of stability is delayed, as in Neishtadt's theory of slowly swept bifurcations.1
• Experiments on an exact quadratic problem, a full-batch MLP and ResNet-32 support the theory; on ResNet-18, much of the apparent benefit of the maps is reproduced by matching the effective learning rate.1

Sources (1)

• [1]Physics-Inspired Optimizers beyond the Edge of Stability: Period-Two Orbits and Delayed Instability
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 09:40 PM

  Gradient descent with momentum can be viewed as a particle moving with friction in the loss landscape.

  In deep learning the learning rate is usually large, so training is a discrete dynamical system: in many full-batch experiments the sharpness grows to the optimizer's stability limit, the edge of stability (EoS), and training continues there.

Extractive summary: sentences quoted from the sources.