Polar Switch: Fast Muon Orthogonalization with Polynomial Circuits

Key points

• Accordingly, a line of research proposes faster polynomial iterations for gradient orthogonalization, such as Polar Express and Gram Newton--Schulz, both built on quintic polynomials.1
• In this paper, we introduce Polar Switch, a new approach to polynomial gradient orthogonalization.1
• Using simple polynomial circuits, Polar Switch reaches a given high polynomial degree at a lower FLOP cost than quintic iterations.1
• Our two recommended Polar Switch methods have 15.5--18.7% lower runtime than Polar Express on square matrices and 8.6--16.2% lower runtime than Gram Newton--Schulz on rectangular matrices with aspect ratios 3.25, 3.5, and 8, with comparable approximation accuracy.1

Sources (1)

• [1]Polar Switch: Fast Muon Orthogonalization with Polynomial Circuits
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:06 PM

  Muon, a 2024-born optimizer for language models, differs from the 2015-born spectral gradient descent mainly in its fast, GPU-friendly Newton--Schulz polynomial for orthogonalizing the gradient matrix.

  Accordingly, a line of research proposes faster polynomial iterations for gradient orthogonalization, such as Polar Express and Gram Newton--Schulz, both built on quintic polynomials.

Extractive summary: sentences quoted from the sources.