From Geometry to Generalization: Why Row Normalization Can Beat Adam and Muon
We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam, a proxy for random-reshuffling Adam, and exact-SVD Muon in high-dimensional multiclass classification.
ProofPaper ↗
Key points
- Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether this difference provably affects population performance remains unclear.
- Under an isotropic Gaussian-cloud data model, this advantage arises because row normalization's class-wise Euclidean geometry asymptotically preserves the population decision-boundary directions, whereas Adam's coordinate-wise geometry and Muon's spectral geometry introduce nonvanishing distortions.
- Beyond isotropy, the advantage persists for full-batch training on class means with independently oriented class-mean and test-noise covariances.
- Synthetic and last-layer language-model experiments support the predicted advantage.
Sources (1)
- [1]From Geometry to Generalization: Why Row Normalization Can Beat Adam and MuonarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 06:15 AM
We show that row-wise normalization can achieve strictly higher population accuracy than full-batch Adam, a proxy for random-reshuffling Adam, and exact-SVD Muon in high-dimensional multiclass classification.
Different optimizers can fit the same training data while selecting classifiers with substantially different geometries, but whether this difference provably affects population performance remains unclear.
Extractive summary: sentences quoted from the sources.