MuPACE: Learning to Pace Muon for LoRA Fine-Tuning

Key points

• Learning rate schedules are typically fixed before training begins, yet when fine-tuning large language models with low-rank adapters (LoRA), the appropriate scale varies substantially with the model, task, and adapter configuration.1
• We instead amortize learning rate selection across LoRA fine-tuning problems.1
• We leave the base optimizer, Muon, untouched, so that its normalized update fixes the direction, and meta-learn only the global step size.1
• Meta-trained only on models with up to 16 layers, MuPACE transfers one-shot to unseen tasks on models as deep as 64 layers, such as Qwen3-32B and Qwen3.8-27B, where a single run remains competitive with swept schedules.1

Sources (1)

• [1]MuPACE: Learning to Pace Muon for LoRA Fine-Tuning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 11:52 AM

  Our method MuPACE is a small recurrent policy, meta-trained across a distribution of fine-tuning runs, that conditions on the observed trajectory and emits bounded adjustments to an interpretable analytic schedule.

  Learning rate schedules are typically fixed before training begins, yet when fine-tuning large language models with low-rank adapters (LoRA), the appropriate scale varies substantially with the model, task, and adapter configuration.

Extractive summary: sentences quoted from the sources.