Hard Way, The Only Way: Provable Merits of Training on Hardest Examples

Key points

• Despite a surge of empirical interest in the "unreasonable effectiveness" of drastic data pruning for AI systems, a unified theoretical understanding of this phenomenon remains elusive.1
• Crucially, we characterize the optimal pruning strategy in the high-sample regime where P greatly exceeds N.1
• We demonstrate that strategic pruning can accelerate the decay of classification test error from the classical sqrt(N/P) rate, typical of vanilla ERM with loss functions without exponential tail (e.g., quadratic loss), to the much faster Bayes-optimal scaling of order N/P.1
• In this setup, we prove that, surprisingly, this optimal performance is achieved by a drastic pruning strategy: discarding almost all training data except for a small proportion, consisting strictly of the 2N hardest training examples.1

Sources (1)

• [1]Hard Way, The Only Way: Provable Merits of Training on Hardest Examples
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 04:00 AM

  In this work, we provide a precise high-dimensional theory of learning under data pruning via weighted empirical risk minimization (ERM).

  Despite a surge of empirical interest in the "unreasonable effectiveness" of drastic data pruning for AI systems, a unified theoretical understanding of this phenomenon remains elusive.

Extractive summary: sentences quoted from the sources.