Weak-$L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

Key points

• In this paper, we consider smooth convex optimization under affine heavy-tailed noise, whose scale may grow with the gradient norm.1
• We derive high-probability convergence guarantees for clipped SGD (Clip-SGD) and its accelerated variant Clip-SSTM under a weak-$L^α$ tail condition, which is strictly weaker than the corresponding moment bound.1
• For $1<α<2$, the clipping bounds have the same powers of the clipping level as in the moment case, and both methods attain the additive stochastic rate $K^{-(α-1)/α}$.1
• We also analyze Clip-SGD with prescribed clipping levels and derive finite-horizon guarantees for reaching a neighborhood determined by the clipping bias.1

Sources (1)

• [1]Weak-$L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 12:18 PM

  Gradient clipping is a key tool for obtaining high-probability convergence guarantees for stochastic first-order methods under heavy-tailed noise.

  In this paper, we consider smooth convex optimization under affine heavy-tailed noise, whose scale may grow with the gradient norm.

Extractive summary: sentences quoted from the sources.