Accelerating Finite-Sum Bilevel Optimization with Fully First-Order Oracles

Key points

• We propose Sampled Implicit Corrections for Bilevel Optimization (SICBO), which combines a penalty approximation with variance-reduced corrections for the lower-level variables.1
• Under componentwise lower strong convexity, SICBO finds a point $x$ satisfying $\mathbb{E}\|\nablaΦ(x)\|^2\le\varepsilon^2$ within $\tilde{O}(N+\sqrt{N}κ^{7/2}\varepsilon^{-2})$ component-gradient calls, where $N$ is the total number of components and $κ$ is the normalized lower-level condition number.1
• The bound achieves square-root finite-sum dependence using only gradients and preserves the best known deterministic fully first-order condition-number exponent for smooth problems.1
• Our improvement follows from a curvature-dependent bound for nonlinear correction errors, which permits epochs of length $\tildeΘ(\sqrt{N/κ})$ with batches of size $\tildeΘ(\sqrt{Nκ})$.1

Sources (1)

• [1]Accelerating Finite-Sum Bilevel Optimization with Fully First-Order Oracles
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 07:58 AM

  We study nonconvex--strongly-convex bilevel optimization with finite-sum objectives and fully first-order oracles.

  We propose Sampled Implicit Corrections for Bilevel Optimization (SICBO), which combines a penalty approximation with variance-reduced corrections for the lower-level variables.

Extractive summary: sentences quoted from the sources.