T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training

Key points

• Muon has recently emerged as a strong optimizer for large language model pretraining by exploiting matrix structure through polar orthogonalization.1
• However, it processes each parameter matrix independently, whereas recent tensor-aware extensions of Muon jointly update groups of eligible matrix parameters through matrix unfoldings.1
• We show that the exact T-MUON update is a steepest-descent direction under the $\starM$-spectral norm and establish a convergence guarantee.1
• Across GPT-2-style and LLaMA-style models from 51M to 1.3B parameters trained on FineWeb and FineWeb-Edu, head-wise T-MUON with the identity transform consistently achieves lower validation perplexity than AdamW, Muon, and variants using DFT, DCT, or DST transforms.1

Sources (1)

• [1]T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 01:20 AM

  In contrast, we propose T-MUON, a t-SVD-based generalization that groups structurally related Transformer gradients into third-order tensors and defines the corresponding Muon-style update directly through the $M$-transform.

  Muon has recently emerged as a strong optimizer for large language model pretraining by exploiting matrix structure through polar orthogonalization.

Extractive summary: sentences quoted from the sources.