Competing Learning in Hybrid Architectures

Key points

• Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention.1
• In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers.1
• Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases.1
• Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language.1

Sources (2)

• [1]Competing Learning in Hybrid Architectures
  Hugging Face Daily Papers · Oct 19, 12:00 AM

  Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval.

  Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention.

• [2]Competing Learning in Hybrid Architectures
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:55 PM · same content

Extractive summary: sentences quoted from the sources.