How Inner Optimizers Shape Replica Scaling

Key points

• Training large language models (LLMs) across geographically distributed and poorly interconnected compute pools or data centers has become a topic of significant interest.1
• This has led to increased study of local optimization methods such as DiLoco for LLMs, especially those that use local (or inner) adaptive or preconditioned optimizers, which are critical for high-performance LLM training.1
• However, a central challenge remains: it is well known that increasing the number of model replicas (replica scaling) while fixing the total training budget leads to monotonically decreasing performance.1
• Our empirical evaluation extends the results to reveal that whitening-based inner-optimizers achieve strong replica scaling (for 8-96 replicas), with performance relative to DP improving as parameters and data are scaled.1

Sources (1)

• [1]How Inner Optimizers Shape Replica Scaling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 07:00 PM

  In this work, we consider a range of inner-optimizers and identify two important factors that impact replica scaling: (1) the covariance structure of the pseudo-gradients and (2) the impact of batch size on the pseudogradient's noise scale.

  Training large language models (LLMs) across geographically distributed and poorly interconnected compute pools or data centers has become a topic of significant interest.

Extractive summary: sentences quoted from the sources.