ResearchResearch paperRetrieval, RAG & Search · Large Language Models1 source · Oct 20, 2026

Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling.

However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content.
Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential.
To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy.
Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.

[1]Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:05 AM
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling.
However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content.

Extractive summary: sentences quoted from the sources.

Before this