Language Guidance for Supervised Vision Training: An Empirical Study of Generalization

Key points

• Deep neural networks have achieved remarkable success on vision benchmarks, yet they continue to struggle with many generalization challenges.1
• Supervised vision training relies on one-hot labels, which provide limited information about semantic structure and shared attributes between classes.1
• Our analyses show that the two mechanisms have complementary strengths, with explicit guidance consistently benefiting in-distribution, low-data performance, and continual learning retention, while implicit guidance is often more useful in shortcut-sensitive settings and under stronger adversarial perturbations.1
• The analyses characterize when language-derived structure helps supervised vision training and provides a practical roadmap for using off-the-shelf pretrained models from another modality as auxiliary supervision.1

Sources (1)

• [1]Language Guidance for Supervised Vision Training: An Empirical Study of Generalization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 07:43 PM

  We study whether pretrained language models can serve as lightweight auxiliary supervision for vision training without requiring paired image-text data, prompt engineering, or contrastive objectives.

  Deep neural networks have achieved remarkable success on vision benchmarks, yet they continue to struggle with many generalization challenges.

Extractive summary: sentences quoted from the sources.