Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

Key points

• Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters.1
• We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of experts that best reconstruct its original output.1
• Exact subset search is combinatorial and becomes intractable for modern MoE models with many experts per layer.1
• Across multiple MoE architectures and tasks spanning mathematical reasoning and code generation, our approach improves over the strongest existing expert-pruning baselines, with absolute accuracy gains of up to 15.05% on Qwen3-30B-A3B, 14.55% on Qwen3-235B-A22B, 11.06% on GLM-4.5-Air, and 16.20% on Mixtral-8x7B.1

Sources (1)

• [1]Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 06:24 PM

  We develop OptEP, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights.

  Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters.

Extractive summary: sentences quoted from the sources.