An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs

Key points

• However, as mixture-of-experts (MoE) architectures, which offer efficient inference at massive parameter scales, gain increasing prevalence in the community, a critical gap arising from the lack of systematic recipes for applying PEFT to MoE models.1
• To narrow this gap, we conduct a large-scale systematic study that investigates a set of essential design recipes, including which modules should be adapted, which granularity should be used to adapt MoE modules, which PEFT method to use, and to what extent should the load-balancing objective be used, across three MoE backbones on downstream tasks and general knowledge tasks.1
• Second, native expert-weight updates offer a better downstream--retention trade-off than adapters that bypass experts or entire MoE modules.1
• We finally evaluate a weight-level redesign of MoE$^2$-LoRA on Qwen3.1

Sources (1)

• [1]An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 01:28 PM

  Parameter-efficient fine-tuning (PEFT) methods are critical for adapting large models under resource-constrained settings.

  However, as mixture-of-experts (MoE) architectures, which offer efficient inference at massive parameter scales, gain increasing prevalence in the community, a critical gap arising from the lack of systematic recipes for applying PEFT to MoE models.

Extractive summary: sentences quoted from the sources.