ResearchResearch paperEfficiency & Inference · Large Language Models1 source · Oct 7, 2026

Optimizing Large Language Models with Chained LMOs

We introduce chained linear minimization oracles (chained LMOs), which cast these methods as compositions of LMOs.

Muon has motivated a growing family of optimizers that compose multiple matrix normalizations, but these methods remain fragmented and lack a unified perspective.
To explain why composition can nevertheless help, we turn to linear associative memory and show that chaining can improve over Muon under anisotropic embeddings.
Empirically, we propose TensorChain, a novel optimizer within the framework that stacks compatible weight matrices across different layers and normalizes the 3d tensor across its axes.
In Qwen3 0.6B and 1.7B pretraining, TensorChain outperforms all chained baselines in average token efficiency, with average token savings of 9.6% over Muon at matched validation loss.

[1]Optimizing Large Language Models with Chained LMOs
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 10:56 PM
We introduce chained linear minimization oracles (chained LMOs), which cast these methods as compositions of LMOs.
Muon has motivated a growing family of optimizers that compose multiple matrix normalizations, but these methods remain fragmented and lack a unified perspective.

Extractive summary: sentences quoted from the sources.

Before this