CAST: History-Adaptive Spectral Clipping for LLM Pretraining
We introduce CAST (Clipping with Adaptive Scales and Thresholds), a history-adaptive extension of SSC.
ProofPaper ↗
Key points
- Soft spectral clipping (SSC) reduces gradient singular values, but its effect depends on the gradient scale and clipping threshold.
- AdamW's previous second moment sets row and column scales.
- An exponential moving average of the clipped output's estimated spectral norm sets each matrix's threshold.
- We train a mixture-of-experts (MoE) model with 19.08B total parameters for 83.89B tokens.
Sources (1)
- [1]CAST: History-Adaptive Spectral Clipping for LLM PretrainingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 08:15 AM
We introduce CAST (Clipping with Adaptive Scales and Thresholds), a history-adaptive extension of SSC.
Soft spectral clipping (SSC) reduces gradient singular values, but its effect depends on the gradient scale and clipping threshold.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026Cortex: Consolidating Context into Parametric Memory
- Oct 14, 2026LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
- Oct 13, 2026Run fast, private agents on-device with LFM2.5-24B-A2B
- Oct 13, 2026Gemini 4 Argon: Built to solve your hardest problems
- Oct 13, 2026Fish Audio Drama 3 Technical Report
- Oct 13, 2026Virtual Head Attention