Data-Efficient Classifier Pretraining Through Diffusion Models

Key points

• Machine learning with limited data is susceptible to overfitting, especially with modern deep neural networks.1
• Prior to the introduction of AlexNet, reconstruction-based pretraining with auto-encoder and denoising auto-encoder were found to be beneficial, partially due to the limited size of the training sets at the time.1
• While contrastive pretraining performs better on CIFAR-10 and Tiny ImageNet in low-data regimes, DiffPT is particularly effective on medical imaging and time-series tasks, where it outperforms self-supervised baselines across a range of low-data regimes.1
• Our results suggest that DiffPT is a simple and effective approach for data-efficient classifier pretraining on different modalities.1

Sources (1)

• [1]Data-Efficient Classifier Pretraining Through Diffusion Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 10:57 AM

  Motivated by these traditional approaches, we propose DiffPT, a simple diffusion modeling approach to pretrain a given classifier backbone for limited domain-specific training data.

  Machine learning with limited data is susceptible to overfitting, especially with modern deep neural networks.

Extractive summary: sentences quoted from the sources.