BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility

Key points

• Muon has emerged as a strong optimizer for neural network training and admits an interpretation as steepest descent under a spectral-norm geometry.1
• Yet we find that Muon-trained Transformers can degrade sharply when their MLP hidden channels are pruned after training.1
• For linear factorization models, we show that Muon admits minimum-cost exact representations with redundant hidden width, whereas under the BMuon oracle geometry and a sufficiently scarce budget, every minimum-cost representation is width-minimal.1
• On ImageNet-1K with ViT-S/16, BMuon improves over Muon under gradual structured MLP pruning, and its learned allocations provide a more effective pruning criterion than weight magnitude.1

Sources (1)

• [1]BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 02:52 PM

  To reconcile Muon's optimization performance with structured compressibility, we introduce Budget Muon (BMuon), which modifies Muon's spectral geometry so that hidden channels across blocks compete for a shared allocation budget.

  Muon has emerged as a strong optimizer for neural network training and admits an interpretation as steepest descent under a spectral-norm geometry.

Extractive summary: sentences quoted from the sources.