Dual-Slot: From Serial Loops to Parallel Slots in Language Models

Key points

• Looped language models increase computational depth by repeatedly applying shared layers, offering a way to improve model quality without adding parameters.1
• However, sequential dependencies between loop iterations increase autoregressive decoding latency.1
• Rather than waiting for the current token's latent computation to finish, the prediction slot combines the previous token's deep latent state with the current representation, while the latent slot produces a state for the next decoding step.1
• To preserve token-level parallelism during training, we use a small number of Jacobi iterations to refine latent states across the sequence in parallel.1

Sources (1)

• [1]Dual-Slot: From Serial Loops to Parallel Slots in Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 07:48 AM

  We introduce Dual-Slot, an architecture that reorganizes serial looped computation into parallel latent and prediction slots.

  Looped language models increase computational depth by repeatedly applying shared layers, offering a way to improve model quality without adding parameters.

Extractive summary: sentences quoted from the sources.