Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion
We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers.
ProofPaper ↗
Key points
- Large language models impose substantial memory demands and arithmetic costs during inference.
- Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs.
- Dense-to-MoE conversion partitions pretrained feed-forward network (FFN) neurons into experts and selects a subset for each token.
- These results establish per-layer expert utilization as a novel optimization axis in dense-to-MoE conversion.
Sources (1)
- [1]Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE ConversionarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 09:48 PM
We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers.
Large language models impose substantial memory demands and arithmetic costs during inference.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 13, 2026Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too
- Oct 13, 2026Update: Converting dense models into Mixture-of-Experts
- Oct 13, 2026inLoRA-MINT: Internal Representations for Auditing Training Data in Domain-adapted LLMs
- Oct 13, 2026Perturbation, Not Provenance: Why Activation Steering Does Not Make a Robust Text Watermark
- Oct 13, 2026Virtual Head Attention
- Oct 13, 2026Beyond the Verdict: Agentic Code Judging with Verifiable Feedback