Global Convergence in Deep Networks via the Second Law of Thermodynamics

Key points

• To bridge this gap, we study $L2$-regularized continuous-time Langevin dynamics (CLD) with isotropic noise, for training feedforward networks with smooth activations.1
• We introduce a change-of-measure inequality, inspired by the second law of thermodynamics, that enables control of key properties of the training dynamics under the time-evolving parameter distribution.1
• Leveraging a sharpened lower bound on the minimum eigenvalue of the Neural Tangent Kernel (NTK) at initialization, we show that injected noise helps preserve NTK stability with high probability without confining the weights to a small neighborhood of their initialization.1
• Our analysis yields explicit conditions on width and noise temperature that guarantee convergence to a finite loss floor.1

Sources (1)

• [1]Global Convergence in Deep Networks via the Second Law of Thermodynamics
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 04:35 PM

  Existing convergence analyses of finite-width overparameterized deep neural networks under standard parameterization often rely on parameters remaining close to their initialization and typically neglect the role of noise, despite empirical evidence that stochasticity can improve optimization and generalization.

  To bridge this gap, we study $L_2$-regularized continuous-time Langevin dynamics (CLD) with isotropic noise, for training feedforward networks with smooth activations.

Extractive summary: sentences quoted from the sources.