One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM

Key points

• Poor generalisation in deep neural networks is frequently associated with convergence to sharp minima.1
• However, SAM relies on a fixed perturbation radius $ρ$ that ignores local curvature, leading to inefficient behaviour: the perturbation may be too small to escape poor basins defined by sharp local minima, or too large to allow convergence to solutions with good loss-sharpness tradeoff.1
• We reinterpret $ρ$ as the intensity of the perturbation signal and propose Curvature-Aware Adaptive Perturbation for SAM (CAP-SAM), a geometry-aware method that modulates the perturbation radius based on a sharpness ratio, enabling a more effective search for robust minima on the loss landscape.1
• We prove that CAP-SAM preserves the same convergence rate as standard SAM in non-convex stochastic settings.1

Sources (1)

• [1]One Size Does Not Fit All: Curvature-Aware Adaptive Perturbation for SAM
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 04:12 PM

  Sharpness-Aware Minimisation (SAM) improves generalisation of deep neural networks by minimising a combination of the training loss and the local sharpness of the loss landscape, biasing convergence towards flat minima.

  Poor generalisation in deep neural networks is frequently associated with convergence to sharp minima.

Extractive summary: sentences quoted from the sources.