Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness

Key points

• Generalized smoothness conditions have recently been proposed in the analysis of gradient methods but remain underexplored for SAM.1
• To analyze the effect of SAM's perturbation under generalized smoothness, we use relative gradient distortion to quantify how the sharpness-aware direction departs from the original gradient.1
• Under $(L0,L1)$-smoothness, we show that this distortion can become unbounded near stationarity for SAM and at large gradients for its unnormalized variant (USAM), with examples where SAM cycles and USAM diverges.1
• Motivated by this condition, we propose Curvature-Scaled SAM (CS-SAM), which scales both the perturbation and the descent step by suitable curvature upper bounds, and prove that it uniformly bounds the distortion.1

Sources (1)

• [1]Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 04:26 AM

  Sharpness-Aware Minimization (SAM) has established itself as a widely used method for training deep neural networks.

  Generalized smoothness conditions have recently been proposed in the analysis of gradient methods but remain underexplored for SAM.

Extractive summary: sentences quoted from the sources.