Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients

Key points

• Large neural networks are commonly trained across many workers, and communication between them is a major cost.1
• Workers therefore take several local optimization steps before sending their net parameter displacements, or pseudo-gradients, to an outer optimizer, which often applies Nesterov momentum.1
• We study how the outer momentum, the number of local steps and the averaging over workers control the training dynamics, and in particular their stability.1
• For random quadratics, we derive an exact mean-square stability criterion in any dimension, which reveals that the inner batch size determines whether increasing the number of local steps eventually stabilizes or destabilizes the outer iteration.1

Sources (1)

• [1]Understanding Stability of Outer Nesterov Momentum on Stochastic Pseudo-Gradients
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 08:23 PM

  Finally, we show that the mean-square dynamics governing stability can be probed directly at intermediate neural-network checkpoints using shared-minibatch perturbations.

  Large neural networks are commonly trained across many workers, and communication between them is a major cost.

Extractive summary: sentences quoted from the sources.