Data Density Scaling Laws for Image Self-Distillation

Key points

• Current scaling law formulations suggest that increasing the number of unique samples in a dataset improves model performance, while repeated exposure to the same samples provides diminishing value.1
• In this work, we examine this assumption for image self-distillation models.1
• Through systematic experiments across diverse datasets, namely Maxar, SATLAS, Waymo, nuPlan, Ego4D, ImageNet, and YFCC15M, we demonstrate that the effects of sample repetition on downstream performance are non-trivial and dataset-dependent.1
• We identify settings where repeatedly training on a smaller random subset can outperform training on a larger set of unique samples.1

Sources (1)

• [1]Data Density Scaling Laws for Image Self-Distillation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 06:08 PM

  To address this gap, we propose a modified version of a Chinchilla-style scaling law that explicitly accounts for two factors: the number of unique samples and repetition per sample.

  Current scaling law formulations suggest that increasing the number of unique samples in a dataset improves model performance, while repeated exposure to the same samples provides diminishing value.

Extractive summary: sentences quoted from the sources.