Improving the Expressive Power of Hybrid Sequence Models via Multi-Hop Retrieval

Key points

• We investigate these questions by analyzing the expressive power of individual hybrid blocks on function composition, a well-studied task in sequence-model expressivity associated with multi-hop reasoning.1
• To overcome this limitation, we introduce the Multi-hop Retrieval Attention (MuRA) hybrid block, a simple architecture that enables multi-hop retrieval from a shared KV cache while preserving asymptotic computational and memory efficiency.1
• We prove that a single fixed size $m$-hop MuRA block can compute $m$-function composition, whereas computing $(m+1)$-function composition with the same retrieval depth requires polynomial model size, establishing a strict expressivity hierarchy in retrieval depth.1
• Together, our results identify multi-hop retrieval as a promising mechanism for improving the expressive power of efficient hybrid architectures.1

Sources (1)

• [1]Improving the Expressive Power of Hybrid Sequence Models via Multi-Hop Retrieval
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 01:10 PM

  Hybrid architectures offer a promising approach to efficient language modeling by combining a small number of full-attention layers with a larger number of efficient sequence layers, such as sparse attention and recurrent layers.

  We investigate these questions by analyzing the expressive power of individual hybrid blocks on function composition, a well-studied task in sequence-model expressivity associated with multi-hop reasoning.

Extractive summary: sentences quoted from the sources.