The Router Already Knows
Mixture-of-experts (MoE) models already make a sparse choice for every token: which experts should process it.
ProofPaper ↗
Key points
- We ask whether a sparse dictionary should reuse that choice.
- NR-SAE gives each routed expert a small dictionary and opens only those the frozen router selects.
- At block 15 of Gemma 4 26B-A4B, after ten million C4 training tokens and across three seeds, the expert-aligned autoencoder explains 0.518 of the variance in router-weighted expert outputs on a fixed evaluation capture.
- At this layer and budget, the router already knows something a dictionary would otherwise have to learn.
Sources (1)
- [1]The Router Already KnowsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 12:17 AM
Mixture-of-experts (MoE) models already make a sparse choice for every token: which experts should process it.
We ask whether a sparse dictionary should reuse that choice.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
- Oct 20, 2026Sage-A1.5: From Long-Horizon Execution to Scientific Exploration with Capability Composition
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models
- Oct 19, 2026MiniMax M3.1 Flash shows China's AI price race has no floor in sight
- Oct 19, 2026NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation
- Oct 19, 2026TorchNeuron: running HF models on AWS Trainium is easier than ever