Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design

Key points

• We isolate these axes in models trained from scratch at matched parameter scale, logical depth, and token budgets, and then test whether the resulting design principles transfer to resident- and active-parameter-matched Mixture-of-Experts (MoE) models.1
• The prelude-recurrent block-coda sandwich architecture consistently improves performance on context extraction and reasoning-intensive tasks.1
• Input injection primarily enhances knowledge retrieval and context utilization, whereas random initialization and separate H/L recurrent states provide mixed or negative returns.1
• At the 8BA0.8B (32BA3B when unrolled) scale trained on 500B tokens, Huginn outperforms Ouro on nearly all benchmarks spanning knowledge, reasoning, and coding, while also exhibiting more balanced expert routing.1

Sources (2)

• [1]Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 07:12 AM

  Looped language models reuse physical Transformer blocks across logical depth, but established architectures entangle recurrence placement, input injection, and recurrent state organization.

  We isolate these axes in models trained from scratch at matched parameter scale, logical depth, and token budgets, and then test whether the resulting design principles transfer to resident- and active-parameter-matched Mixture-of-Experts (MoE) models.

• [2]Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design
  Hugging Face Daily Papers · Oct 16, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.