Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking

Key points

• While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions.1
• We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions.1
• We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions.1
• Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure.1

Sources (1)

• [1]Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 04:06 AM

  Flat loss landscapes have long been linked to better generalization in neural networks.

  While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions.

Extractive summary: sentences quoted from the sources.