Near-Optimal Last-Iterate Convergence of SGD in the Smooth Interpolation Regime

Key points

• Our analysis is based on a new technique, which compares the second moments of the stochastic gradients along the SGD trajectory across only logarithmically many time scales.1
• For quadratic losses in the interpolation regime, an adaptation of this technique gives an $O(1/T)$ rate for every constant stepsize, extending the recent optimal rate for the randomized Kaczmarz method of Derezinski and Dong (2026) to stepsizes above the greedy stepsize with an arguably simpler proof.1

Sources (1)

• [1]Near-Optimal Last-Iterate Convergence of SGD in the Smooth Interpolation Regime
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 06:53 PM

  We study the last iterate of stochastic gradient descent (SGD) with a constant stepsize on smooth convex objectives in the low-noise and interpolation regimes, where the variance $σ_\star^2$ of the stochastic gradients at the optimum is small or zero.

  Our analysis is based on a new technique, which compares the second moments of the stochastic gradients along the SGD trajectory across only logarithmically many time scales.

Extractive summary: sentences quoted from the sources.