Newton-Based Methods for Bilevel Optimization Beyond Lower-Level Strong Convexity

Key points

• Bilevel optimization provides a framework for machine learning tasks such as hyperparameter optimization, data reweighting, and meta-learning.1
• Existing Newton-based bilevel methods typically rely on the classical hypergradient formula, which is generally unavailable when lower-level solutions are nonunique and the lower-level Hessian is singular.1
• Newton-based BAMM uses a strongly convex aggregation model when the lower-level objective is convex and the upper-level objective is strongly convex in the lower-level variable.1
• We further introduce GNBO, a generalized Newton-based framework that captures the shared-curvature structure of the proposed methods and several existing Newton and quasi-Newton bilevel methods.1

Sources (1)

• [1]Newton-Based Methods for Bilevel Optimization Beyond Lower-Level Strong Convexity
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:58 PM

  We develop two Newton-based methods beyond lower-level strong convexity by exploiting curvature shared by the lower-level and multiplier updates.

  Bilevel optimization provides a framework for machine learning tasks such as hyperparameter optimization, data reweighting, and meta-learning.

Extractive summary: sentences quoted from the sources.