SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs
We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data.
ProofPaper ↗
Key points
- Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment.
- Expert pruning reduces this footprint, but most methods focus on which experts to remove and prune every layer equally.
- A perturbation bound under random pruning formalizes this intuition, and controlled interventions with a fixed selector link concentrated routing to larger output changes.
- With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios.
Sources (1)
- [1]SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 12:40 PM
We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data.
Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Agnes-AI/Agnes-3.0-Qwen (QFN finetune)
- Oct 20, 2026Voxtral 3
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 19, 2026Amazon SageMaker JumpStart: 2026 year-to-date in review
- Oct 19, 2026Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
- Oct 19, 2026inclusionAI/Ming-3.0-audio-tiny · Hugging Face