Adaptive Variance Reduction without Knowing Oracle Smoothness

Key points

• Existing adaptive variance reduction methods are especially designed for average smoothness and do not adapt to standard smoothness.1
• In this paper, we study how one algorithm can adapt to these two settings and the noiseless setting without knowing the smoothness constants or the variance bound.1
• Given the gradient budget $T$, we develop a two-phase method.1
• When $T$ is unknown, we propose a method with two paths that attain the corresponding rates with logarithmic factors in the stochastic settings.1

Sources (1)

• [1]Adaptive Variance Reduction without Knowing Oracle Smoothness
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 04:13 PM

  Variance reduction improves the convergence rate of stochastic non-convex optimization under the average smoothness condition.

  Existing adaptive variance reduction methods are especially designed for average smoothness and do not adapt to standard smoothness.

Extractive summary: sentences quoted from the sources.