Block Recursive Transformers for Structured Parameter Sharing

Key points

• Existing static schemes adopt two extreme reuse topologies: layer-wise repetition, where each layer is reused independently, and global cyclic reuse, where the entire stack is repeatedly applied as a single shared function.1
• We argue that sharing granularity is an important factor in recursive parameter sharing.1
• This study proposes Block Recursive Transformers (BRT), which partition layers into contiguous blocks and recursively reuse each block before moving to the next.1
• Across decoder-only models up to 1.7B parameters, with additional encoder-decoder and encoder-only studies, BRT improves validation negative log-likelihood and downstream accuracy over static recursive sharing baselines.1

Sources (1)

• [1]Block Recursive Transformers for Structured Parameter Sharing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 04:02 AM

  Recursive Transformers depend not only on how many parameters are shared, but also on how shared parameters are organized across depth.

  Existing static schemes adopt two extreme reuse topologies: layer-wise repetition, where each layer is reused independently, and global cyclic reuse, where the entire stack is repeatedly applied as a single shared function.

Extractive summary: sentences quoted from the sources.