On the Last-Iterate Convergence of Clipped Gradient Methods

Key points

• Notably, people have demonstrated that one of the most famous, simple, and popular methods, Stochastic Gradient Descent ($\mathtt{SGD}$), provably achieves last-iterate convergence.1
• To address the challenge of heavy-tailed noise, people have proposed $\mathtt{Clipped}-\mathtt{SGD}$, an algorithm that combines $\mathtt{SGD}$ with a simple mechanism, gradient clipping.1
• Although $\mathtt{Clipped}-\mathtt{SGD}$ performs well in practice, its last-iterate convergence property remains highly underexplored.1
• In this work, we prove the first optimal last-iterate rates in high probability for $\mathtt{Clipped}-\mathtt{SGD}$ under heavy-tailed noise, thereby closing a gap in the literature.1

Sources (1)

• [1]On the Last-Iterate Convergence of Clipped Gradient Methods
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 11:46 PM

  In stochastic (strongly) convex optimization, establishing last-iterate convergence guarantees for first-order methods has recently garnered increasing attention, as this not only deepens the theoretical understanding of these algorithms but also aligns more closely with practice.

  Notably, people have demonstrated that one of the most famous, simple, and popular methods, Stochastic Gradient Descent ($\mathtt{SGD}$), provably achieves last-iterate convergence.

Extractive summary: sentences quoted from the sources.