Variance Reduction with Delayed Feedback for Non-Convex Optimization

Key points

• We first analyze an ordered recursive momentum estimator for stochastic optimization.1
• Under average smoothness and bounded variance, its oracle complexity is $O(1+σ^2ε^{-2}+LΔf(1+σ/ε+τ)ε^{-2})$, where the target is an expected squared gradient norm of at most $ε^2$ and $τ$ bounds the feedback delay.1
• An asynchronous snapshot variant continues updating during full-gradient computations under an explicit bound on their completion time.1
• For the stochastic method, we further replace a maximum-delay bound by a bound on the weighted amount of missing feedback.1

Sources (1)

• [1]Variance Reduction with Delayed Feedback for Non-Convex Optimization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:48 PM

  We study variance reduction for non-convex optimization when gradient evaluations return with a delay.

  We first analyze an ordered recursive momentum estimator for stochastic optimization.

Extractive summary: sentences quoted from the sources.