Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs

Key points

• We classify four representative prior methods into a unified taxonomy and benchmark them against a natural, simple baseline: statically reducing the number of experts $k$ (i.e., same reduced $k$ across all layers and tokens).1
• We provide a fair comparison of these methods at matched expert-compute budget, on two modern MoEs (Gemma4-26B-A4B and Qwen3-30B-A3B) in a text-only setting and a contemporary vision-language model (Qwen3-VL) in a multimodal setting.1
• Surprisingly, we find that the simple baseline is hard to outperform consistently across models, modalities, benchmarks, and calibration settings.1
• We further find that the effective $k$ achieved (or optimized for) in calibration can diverge from the effective $k$ in testing, resulting in over- or under-estimating the compute at inference time.1

Sources (1)

• [1]Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 03:29 PM

  A growing body of work proposes dynamic expert-skipping methods for Mixture-of-Experts (MoE) models that adjust the number of experts activated per token during inference, aiming to preserve model quality while further reducing the number of active parameters.

  We classify four representative prior methods into a unified taxonomy and benchmark them against a natural, simple baseline: statically reducing the number of experts $k$ (i.e., same reduced $k$ across all layers and tokens).

Extractive summary: sentences quoted from the sources.