BatchParScale: Sparse Batch-Mixing for Sequence Models

Key points

• ParScale takes a step toward model-internal parallel scaling by learning to aggregate multiple forward passes, but its streams still do not exchange information while processing a sequence.1
• We introduce BatchParScale, an attention mechanism that allows parallel streams to exchange information through a small set of communication positions.1
• We evaluate BatchParScale on a new Parallel Needle-in-a-Haystack retrieval benchmark and in pretraining, fine-tuning, and cooperative multi-agent reinforcement learning.1
• At the same time, fact retrieval performance for pretrained and finetuned BatchParScale models remains unclear.1

Sources (1)

• [1]BatchParScale: Sparse Batch-Mixing for Sequence Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 01:14 PM

  Recent test-time scaling methods improve model performance by increasing inference-time compute.

  ParScale takes a step toward model-internal parallel scaling by learning to aggregate multiple forward passes, but its streams still do not exchange information while processing a sequence.

Extractive summary: sentences quoted from the sources.