Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation
Fine-grained mixture-of-experts (MoE) models commonly increase expert count by narrowing each expert, treating expert shape as a by-product of routing granularity.
ProofPaper ↗
Key points
- We show that an expert's local Jacobian rank is jointly bounded by these two dimensions, and that balancing them maximizes the rank ceiling at fixed per-expert parameters.
- Controlled shape sweeps across model scales and routing granularities consistently favor near-square experts.
- We further use layerwise intrinsic dimension (ID) to guide capacity allocation across depth, co-varying the number and shape of experts while keeping per-layer parameter count, activated FLOPs, and communication volume constant.
- These results establish expert shape and layerwise allocation as practical design axes for MoEs.
Sources (1)
- [1]Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity AllocationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 06:54 AM
Fine-grained mixture-of-experts (MoE) models commonly increase expert count by narrowing each expert, treating expert shape as a by-product of routing granularity.
We show that an expert's local Jacobian rank is jointly bounded by these two dimensions, and that balancing them maximizes the rank ceiling at fixed per-expert parameters.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026Overcoming Data Imbalances with Expert Selection Biases
- Oct 9, 2026DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving
- Oct 9, 2026Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
- Oct 9, 2026Introducing Clef-omni with full multimodality, plus a faster Clef and a cheaper Clef-flash
- Oct 9, 2026TAME-Nav: Temporal Asymmetric Mixture-of-Experts Distillation for Agile and Versatile Robot Navigation
- Oct 9, 2026Multi-Stream Attention Residuals