Hybrid Momentum Methods for Optimizing Deep Learning Models
We introduce Hybrid Momentum (HyM), which combines a linearly weighted average of past gradients with a geometrically discounted fast momentum.
ProofPaper ↗
Key points
- Momentum is central to stochastic optimization in deep learning, including the training of large language models.
- In particular, we propose HyMUON, which applies Newton--Schulz orthogonalization to the hybrid direction.
- Across image classification, machine translation, and language modeling, HyM and HyMUON match or improve upon strong baselines.
- For LLaMA-1.3B pretraining on C4, HyMUON achieves a final validation loss of 2.777, compared with 2.903 for Muon.
Sources (1)
- [1]Hybrid Momentum Methods for Optimizing Deep Learning ModelsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 05:39 AM
We introduce Hybrid Momentum (HyM), which combines a linearly weighted average of past gradients with a geometrically discounted fast momentum.
Momentum is central to stochastic optimization in deep learning, including the training of large language models.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models
- Oct 19, 2026Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
- Oct 19, 2026LIRQuant: An Efficient Locally Isolated Rotation Quantization Method for Large Language Models
- Oct 18, 2026T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
- Oct 18, 2026StopLoss: Teaching Agents to Predict Failure and Stop Early
- Oct 17, 2026A Statistical Viewpoint on Adam, Shampoo and Muon