AdamPM: Memory-Efficient LLM Optimization via Moment Pooling

Key points

• We propose AdamPM (Adam with Pooled Moments), a simple and memory-efficient variant of Adam that groups adjacent moment entries along the row or column dimension and represents each group by a single shared mean.1
• We systematically investigate pooling of both first and second moment estimates and identify second moment pooling as the effective design.1
• With a group size of $s$, AdamPM reduces second moment memory to $1/s$ of the original requirement.1
• Extensive experiments on language model pretraining, fine-tuning and multimodal learning demonstrate that AdamPM maintains model performance and training speed while substantially reducing optimizer-state memory.1

Sources (1)

• [1]AdamPM: Memory-Efficient LLM Optimization via Moment Pooling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 10:04 AM

  Adam is widely used for training large language models (LLMs) owing to its strong optimization performance and training stability, but its element-wise estimation for the first and second moments incur substantial memory overhead.

  We propose AdamPM (Adam with Pooled Moments), a simple and memory-efficient variant of Adam that groups adjacent moment entries along the row or column dimension and represents each group by a single shared mean.

Extractive summary: sentences quoted from the sources.