The Ephemeral Cost of Switching Optimizers Across LLM Training Stages

Key points

• Recent work suggests that switching optimizers between pretraining and subsequent training stages incurs a performance cost, but systematic investigation of this phenomenon is lacking.1
• Surprisingly, we find little evidence of a meaningful optimizer mismatch penalty: final performance is largely explained by the independent effects of the pretraining and post-training optimizers, with downstream performance generally improving as the proportion of Muon updates during SFT increases.1
• Although matching the pretraining optimizer can reduce forgetting during SFT, this does not translate into improved final benchmark performance.1
• We further evaluate AdamW and Muon for task-general SFT of open-weight models, across the Llama and Qwen families up to 14B parameters, where Muon remains competitive with AdamW, despite these models having been pretrained with AdamW.1

Sources (1)

• [1]The Ephemeral Cost of Switching Optimizers Across LLM Training Stages
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 09:10 PM

  We study the mismatch between AdamW and Muon optimizers by pretraining 1.2B Llama-3 models from scratch while varying the proportion of updates taken with each optimizer, and similarly interpolating between the two optimizers during continued pretraining and supervised fine-tuning (SFT).

  Recent work suggests that switching optimizers between pretraining and subsequent training stages incurs a performance cost, but systematic investigation of this phenomenon is lacking.

Extractive summary: sentences quoted from the sources.