ResearchResearch paperLarge Language Models · Training & Scaling2 sources · Oct 14, 2026

LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets.

To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit.
This asymmetry motivates us to explore asymmetric compute allocation.
We propose LatentLoop, which compresses the first visit's sequence, retains full resolution in the second, and reallocates saved computation to increase the hidden size.
These results suggest asymmetric compute allocation is a promising direction for looped MoE models.

[1]LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Hugging Face Daily Papers · Oct 14, 12:00 AM
Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets.
To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit.
[2]LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 10:57 AM · same content

Extractive summary: sentences quoted from the sources.

Before this