Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates.
ProofPaper ↗
Key points
- We redesign 4-bit optimizer-state quantization for AdamW from the perspective of rounding space: the coordinate in which a quantizer chooses between adjacent reconstruction levels.
- For the second moment, a local analysis of the quantization cell adjacent to zero shows that small mean state error need not imply small mean preconditioner error at the next step.
- A one-dimensional quadratic construction further shows qualitatively different optimization dynamics under state-space and preconditioner-space rounding.
- These results motivate Zero-Inclusive Preconditioner-space Stochastic Rounding (ZIP-SR), which retains zero in the second-moment codebook and computes stochastic-rounding probabilities in preconditioner space.
Sources (1)
- [1]Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State QuantizationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 05:58 PM
Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates.
We redesign 4-bit optimizer-state quantization for AdamW from the perspective of rounding space: the coordinate in which a quantizer chooses between adjacent reconstruction levels.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 8, 2026Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review
- Oct 7, 2026Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models
- Oct 7, 2026OrBIT: Structure-Guided Embedding Compression
- Oct 7, 2026Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts
- Oct 7, 2026Beyond Anonymous Captions: Grounding Character Identity in Video Captioning and Question Answering
- Oct 7, 2026Successive Training Stages and Large Language Model Persuasion: Effects of Misalignment, Supervised Fine-Tuning, and Preference Optimization