Assessing Representation Semantics Under Language Pretraining in Fine-Grained Classification

Key points

• We propose a compact explainability-driven framework for comparing encoders trained with classification-only and caption-generation supervision on CUB-200.1
• Using a controlled two-stage protocol in which both encoders are ultimately optimized for the same classification task, we enable a direct comparison of the resulting representations.1
• Saliency maps are generated using attention rollout and gradient-weighted rollout and evaluated with five metrics covering faithfulness (positive and negative perturbation) and localization (keypoint emphasis, background attribution, and Earth Mover's Distance).1
• Across random seeds and explanation methods, caption supervision yields representations that are more sensitive to the removal of salient regions, better aligned with annotated object parts, and less reliant on background regions, whereas classification supervision produces representations that are slightly more robust to the removal of low-saliency regions.1

Sources (1)

• [1]Assessing Representation Semantics Under Language Pretraining in Fine-Grained Classification
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 11:39 AM

  Understanding how supervision signals shape semantic visual representations remains challenging, particularly in fine-grained, low-data settings.

  We propose a compact explainability-driven framework for comparing encoders trained with classification-only and caption-generation supervision on CUB-200.

Extractive summary: sentences quoted from the sources.