Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective

Key points

• Under whitened inputs and isotropic imbalance, we combine a closed-form solution of the dynamics with prior random matrix theory results to derive a high-dimensional limit for the dynamics.1
• Our method yields exact asymptotic spectral distributions for the weight matrices at all times and characterizes the evolution of the learned subspace.1
• In the small-initialization limit, we identify a silent alignment phenomenon: the singular directions of the network align on an order-one timescale, while the singular values and empirical loss remain at their initial values.1
• Visible learning occurs later, on a logarithmic timescale, as singular values progressively escape zero and the loss decreases.1

Sources (1)

• [1]Silent Alignment in Two-Layer Linear Networks: A High-Dimensional Perspective
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 03:20 PM

  We study gradient flow in a two-layer linear network in the high-dimensional regime where the input, inner and output dimensions diverge proportionally.

  Under whitened inputs and isotropic imbalance, we combine a closed-form solution of the dynamics with prior random matrix theory results to derive a high-dimensional limit for the dynamics.

Extractive summary: sentences quoted from the sources.