Over-parameterization Is Neither Robust nor Fragile in Matrix Sensing: A Geometric Theory of Solution Robustness

Key points

• We study this question in matrix sensing, a canonical non-convex problem that can also be viewed, under an appropriate parameterization, as training a two-layer quadratic neural network.1
• We develop a rank-aware robustness theory for the Burer-Monteiro factorization when the search rank k may exceed the target rank rstar, thereby capturing over-parameterization.1
• Our analysis shows that robustness depends jointly on the degree of over-parameterization and the local geometry of the attained second-order stationary point: increasing k generally worsens sensitivity, but stationary points with the same search rank can exhibit substantially different robustness.1
• Motivated by this observation, we propose a trajectory-aware algorithm that uses spectral balancing and randomized sketching to steer optimization toward less sensitive solutions.1

Sources (1)

• [1]Over-parameterization Is Neither Robust nor Fragile in Matrix Sensing: A Geometric Theory of Solution Robustness
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 01:00 PM

  Over-parameterization is ubiquitous in modern machine learning, yet its effect on robustness remains poorly understood.

  We study this question in matrix sensing, a canonical non-convex problem that can also be viewed, under an appropriate parameterization, as training a two-layer quadratic neural network.

Extractive summary: sentences quoted from the sources.