Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning

Key points

• Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law.1
• First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift.1
• Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent.1
• Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent.1

Sources (1)

• [1]Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 11:41 AM

  We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained source model is trained via ridge regression, and (ii) a fine-tuned target model is regularized via the $L_2$ distance from the source estimator.

  Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law.

Extractive summary: sentences quoted from the sources.