Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder

Key points

• Looped Transformers increase effective depth by repeatedly applying shared layers, offering a way to scale computation without increasing parameter count.1
• However, full-stack looping places context construction and iterative state refinement within the same recurrent computation, without explicitly distinguishing their roles.1
• A single-pass causal encoder produces layer-specific Global key-value (KV) caches that remain fixed across iterations, while a weight-tied recurrent decoder generates Local KV from its evolving state.1
• For training, we apply a heavy-tail-guided layerwise learning-rate method to the encoder and decoder.1

Sources (1)

• [1]Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:36 PM

  We introduce the Asymmetric Looped Transformer (AsymLoop), which separates single-pass context encoding from recurrent state refinement.

  Looped Transformers increase effective depth by repeatedly applying shared layers, offering a way to scale computation without increasing parameter count.

Extractive summary: sentences quoted from the sources.