Fine-Grained Units, Flat Decisions: Hierarchical Micro-Expert Pruning for Mixture-of-Experts Models

Key points

• However, finer pruning granularity alone does not guarantee the preservation of downstream capabilities, as similar language-modeling perplexity can accompany substantially different task performance.1
• We revisit micro-expert mask construction as two distinct decisions: allocating retained capacity across experts and selecting channels within that capacity.1
• We propose UC-RAMP, a two-stage post-training pruning framework that explicitly separates these decisions.1
• Across three MoE models at 25% and 50% routed-expert channel pruning, UC-RAMP achieves the highest ten-task mean accuracy among evaluated pruning methods in five of six settings.1

Sources (1)

• [1]Fine-Grained Units, Flat Decisions: Hierarchical Micro-Expert Pruning for Mixture-of-Experts Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:10 AM

  Micro-expert pruning reduces the capacity of mixture-of-experts (MoE) models by removing coupled channels within routed experts.

  However, finer pruning granularity alone does not guarantee the preservation of downstream capabilities, as similar language-modeling perplexity can accompany substantially different task performance.

Extractive summary: sentences quoted from the sources.