Stochastic Gradient Descent: A Nonlinear Time Series Perspective
This paper investigates the statistical behavior of Stochastic Gradient Descent (SGD) with constant step size under the framework of iterated random functions.
ProofPaper ↗
Key points
- Compared to previous studies establishing the convergence of SGD in probability measure, e.g., Wasserstein distance, our approach directly provides the convergence in Euclidean distance by showing the Geometric Moment Contraction (GMC) of SGD.
- Specifically, we prove quenched central limit theorem, invariance principle and $α$-stable convergence for averaged SGD (ASGD) regardless of initialization.
- The existence of stationary solutions for the derivative processes is shown, enabling the use of interpolation to recover the full-trajectory of SGD.
- Finally, we propose a new online inference method and a bias-reduced variant for the extrapolated ASGD.
Sources (1)
- [1]Stochastic Gradient Descent: A Nonlinear Time Series PerspectivearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 08:22 PM
This paper investigates the statistical behavior of Stochastic Gradient Descent (SGD) with constant step size under the framework of iterated random functions.
Compared to previous studies establishing the convergence of SGD in probability measure, e.g., Wasserstein distance, our approach directly provides the convergence in Euclidean distance by showing the Geometric Moment Contraction (GMC) of SGD.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026Long-Context Drafting Does Not Require Long-Context Drafters
- Oct 14, 2026When Reasoning Traces Conflict with Evidence: How Answer Configuration Shapes Evidence Following
- Oct 14, 2026What Do Foundation Models for Market Microstructure Learn? A Controlled Study with Synthetic Limit Order Books
- Oct 14, 2026mistralai/Timestral-209M
- Oct 14, 2026LEV: Thinking Fast and Deep with Looped Decision Models
- Oct 13, 2026Correction Training for Diffusion Models