A Statistical Viewpoint on Adam, Shampoo and Muon
We provide new statistical insights into designing stochastic optimization methods.
ProofPaper ↗
Key points
- By modeling the gradient via a normal distribution, we argue that the stochastic gradient should be whitened using its covariance.
- We also recover Muon by applying this same statistical framework to the gradient direction rather than the full gradient.
- Our experiments on GPT-2-type models at Chinchilla scale indicate that MND, the diagonal implementation of MN, outperforms both the Distributed Shampoo and the KL-Shampoo implementations in terms of stability, final validation loss and time taken.
- MND-Muon, the diagonal implementation of MN-Muon, improves on Muon in 21 of 24 paired GPT-small runs and achieves a lower best validation loss on GPT-medium, where it remains stable over the six tested learning rates.
Sources (1)
- [1]A Statistical Viewpoint on Adam, Shampoo and MuonarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 07:41 PM
We provide new statistical insights into designing stochastic optimization methods.
By modeling the gradient via a normal distribution, we argue that the stochastic gradient should be whitened using its covariance.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 17, 2026model : add Kolibri-1 (Aleph-Alpha) support by riteme · Pull Request #30673 · ggml-org/llama.cpp
- Oct 16, 2026Best of Both Worlds: Fine-Tuned Small Models Complement LLMs for Cost and Performance Benefits
- Oct 16, 2026Reasoning with Context Management and External Memory
- Oct 16, 2026BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility
- Oct 16, 2026Towards Hyperparameter Transfer for Differentially Private Optimization
- Oct 15, 2026Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint