Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry

Key points

• Muon controls weight updates through the RMS movement of a linear layer's output.1
• Hilbert distance bounds changes in cross-entropy loss and limits probability redistribution.1
• For scaled dot-product attention, the same principle yields the first-order query--key steepest descent problem underlying Compositional Muon \citep{keigwin2026compositional}.1
• These connections motivate using Hilbert distance as a guiding principle for optimizer design in softmax modules.1

Sources (1)

• [1]Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 02:07 AM

  We propose Hilbert distance as a natural metric for bounding changes in these output distributions.

  Muon controls weight updates through the RMS movement of a linear layer's output.

Extractive summary: sentences quoted from the sources.