Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

Key points

• On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less.1
• In this work, we introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions.1
• SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods.1
• Overall, SST advances the learning--forgetting Pareto frontier, making it a practical default for fine-tuning pre-trained language models.1

Sources (1)

• [1]Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:44 PM

  Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent works connect forgetting to how far, and along which directions, fine-tuning moves the weights.

  On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less.

Extractive summary: sentences quoted from the sources.