Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts
Mixture-of-Experts (MoE) models combine specialized experts through an input-dependent routing mechanism, and their sparse variants underpin many modern large-scale systems.
ProofPaper ↗
Key points
- We study the estimation error of least-squares MoE estimators when both the experts and the gating network are Lipschitz in their parameters.
- Using its empirical representation, we derive an oracle inequality with a fast rate of order $\tilde{\mathcal O}\big((Kd{pr,e}+d{pr,g})/n\big)$ for dense MoEs with $K$ experts.
- For sparse Top-$k$ and Top-$p$ routing, the selection map is discontinuous in the gating parameters.
- We show that such a routing-complexity condition cannot be dropped: Lipschitz continuity in the parameters alone does not suffice.
Sources (1)
- [1]Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of ExpertsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 04:24 PM
Mixture-of-Experts (MoE) models combine specialized experts through an input-dependent routing mechanism, and their sparse variants underpin many modern large-scale systems.
We study the estimation error of least-squares MoE estimators when both the experts and the gating network are Lipschitz in their parameters.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 10, 2026Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation
- Oct 9, 2026Overcoming Data Imbalances with Expert Selection Biases
- Oct 9, 2026DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving
- Oct 9, 2026Introducing Clef-omni with full multimodality, plus a faster Clef and a cheaper Clef-flash
- Oct 9, 2026Multi-Stream Attention Residuals
- Oct 8, 2026The Epoch Brief - October 8, 2026