From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias

Key points

• In practice, however, Muon approximates orthogonalization using finite Newton-Schulz iterations and a positive stabilization constant.1
• We prove that these implementation choices cause Muon to select the Frobenius rather than the spectral max-margin direction in linear classification with separable data.1
• We then disentangle the roles of Newton-Schulz iteration count, the stabilization constant, and the Newton-Schulz polynomial, revealing distinct asymptotic behaviors, including directions that achieve neither Frobenius nor spectral max-margin.1
• Finally, we show that Muon can recover its spectral max-margin bias with Newton-Schulz iterations growing logarithmically over training.1

Sources (1)

• [1]From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 12:42 PM

  Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (2025) established spectral max-margin behavior for spectral-norm steepest descent methods, including idealized Muon with exact orthogonalization.

  In practice, however, Muon approximates orthogonalization using finite Newton-Schulz iterations and a positive stabilization constant.

Extractive summary: sentences quoted from the sources.