Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics

Key points

• In this work, we study such spectral optimizers with a constant step size on strongly convex matrix quadratics, which isolate the effect of the update from the confounds of nonconvexity, noise, and step size schedules.1
• We analyze an update using two quantities: the gain, which is the factor by which it amplifies the gradient along each singular direction, and the length, which is the norm of the update.1
• Interestingly, the practical variants of Muon fall in all three regimes, depending only on how the update is normalized and scaled and not on the polynomial that approximates the polar factor.1

Sources (1)

• [1]Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 07:07 AM

  Muon and related optimizers update a weight matrix with a spectrally transformed gradient.

  In this work, we study such spectral optimizers with a constant step size on strongly convex matrix quadratics, which isolate the effect of the update from the confounds of nonconvexity, noise, and step size schedules.

Extractive summary: sentences quoted from the sources.