Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

Key points

• We first revisit infinite-compute pretraining (Kim et al. 2026) and show that its frontier depends critically on choices previously fixed or studied only in isolation.1
• In particular, optimizing batch size together with epoch counts reveals a U-shaped relationship between validation loss and batch size, favoring smaller batches than typical pretraining, likely because small-batch gradient noise provides implicit regularization.1
• We further find that training recipes, previously optimized under a single-pass assumption, can be tuned, e.g., repacking documents into new sequences every epoch, weight tying, and positive embedding weight decay.1
• Nevertheless, even under a fixed time budget, a smaller batch with fewer GPUs can achieve lower validation loss than a larger batch with more GPUs, challenging the common practice of increasing data parallelism to maximize GPU utilization under a deadline.1

Sources (2)

• [1]Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 10:42 PM

  We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data.

  We first revisit infinite-compute pretraining (Kim et al. 2026) and show that its frontier depends critically on choices previously fixed or studied only in isolation.

• [2]Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs
  Hugging Face Daily Papers · Oct 11, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.