Ori: Drift-Aware Streaming $k$-PCA for Very Low-Rank LLM Training

Key points

• A prominent family of low-rank LLM optimizers, pioneered by GaLore, performs gradient-subspace projection: at each step, they project the gradient into a low-dimensional subspace and map the update back to the full model.1
• In this work, we explicitly identify in practice the slow, stable manner in which LLM gradient subspaces drift across time.1
• Ori employs a streaming $k$-PCA algorithm inspired by Oja++, whose low per-step cost enables high-frequency, on-the-fly subspace tracking.1
• In practice, Ori reduces the performance gap of the state-of-the-art low-rank optimizers to full-rank optimizers by up to 69.3% on a 1B LLaMA pretraining task, and achieves the best performance on 10 out of 16 task scenarios in Roberta-Base fine-tuning.1

Sources (1)

• [1]Ori: Drift-Aware Streaming $k$-PCA for Very Low-Rank LLM Training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 01:13 AM

  To this end, we propose Ori, a novel low-rank optimizer for training LLMs that captures the shifting principal gradient directions over time to achieve state-of-the-art accuracy under stringent low-rank constraints.

  A prominent family of low-rank LLM optimizers, pioneered by GaLore, performs gradient-subspace projection: at each step, they project the gradient into a low-dimensional subspace and map the update back to the full model.

Extractive summary: sentences quoted from the sources.