Mesoscopic learning dynamics of extensive-width neural networks

Key points

• We study this question in two-layer neural networks learning target functions represented by networks of the same architecture, with arbitrary nonlinearity, and with network width and input dimension growing proportionally.1
• Here, we develop a mesoscopic description of learning by characterizing the distribution of overlaps.1
• In the early stages of learning, we find that a Gaussian description of overlap statistics reduces the learning dynamics to a few collective variables that track learning of the target function's linear and quadratic components.1
• We find that specialization in extensive-rank networks is accelerated via a quadratic shielding mechanism, whereby learning the linear and quadratic components of the target function filters gradient fluctuations.1

Sources (1)

• [1]Mesoscopic learning dynamics of extensive-width neural networks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 05:12 PM

  Understanding how stochastic gradient descent learns internal representations in wide neural networks trained on rich target functions is a central problem in machine learning.

  We study this question in two-layer neural networks learning target functions represented by networks of the same architecture, with arbitrary nonlinearity, and with network width and input dimension growing proportionally.

Extractive summary: sentences quoted from the sources.