Dimension-Free Scaling Laws for Size Generalization: An Exponential Separation

Key points

• Modern machine learning models increasingly operate on variable-size structured data, such as sets and graphs.1
• Although extensive work has established when models on sets and graphs can be transferred across input sizes, such results do not characterize the statistical cost of doing so: the sample complexity of size generalization, and its relationship to classical in-domain learning, remain largely unknown.1
• We develop a statistical learning theory of size generalization and establish a sharp exponential separation in sample complexity.1
• To our knowledge, this work provides the first dimension-free minimax theory of size generalization and a rigorous characterization of when, and at what statistical cost, learning can transfer across input sizes.1

Sources (1)

• [1]Dimension-Free Scaling Laws for Size Generalization: An Exponential Separation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 10:55 PM

  Letting $n$ denote the number of training samples, we show that polynomial source decay is insufficient for efficient size generalization: while it yields error $n^{-Θ(1)}$ and hence polynomial sample complexity for in-domain learning, size-generalization error decays only as $(\log n)^{-Θ(1)}$, yielding an exponential separation.

  Modern machine learning models increasingly operate on variable-size structured data, such as sets and graphs.

Extractive summary: sentences quoted from the sources.