The Explainability Illusion: Must ViTs Exhibit Patch-Decoding Interpretability?

Key points

• Transformers are widely assumed to exhibit token-decoding explainability.1
• In particular, in vision transformer (ViT) classifiers, decoding individual patch tokens has become a common diagnostic tool due to its tendency to yield human-interpretable outputs.1
• In this paper we systematically study token-decoding explainability across diverse ViT settings, spanning architectures, scales, and training objectives, using both controlled experiments and real-world models.1
• We provide theoretical support for these observations by demonstrating a setting where a single fixed toy transformer architecture can provably implement the optimal classification rule in multiple ways, which differ in their patch-decoding explainability patterns.1

Sources (1)

• [1]The Explainability Illusion: Must ViTs Exhibit Patch-Decoding Interpretability?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 02:10 PM

  We find that while some models indeed exhibit aligned and interpretable token decodings, others exhibit either inverted explainability maps or a complete lack of correlation with the image semantics.

  Transformers are widely assumed to exhibit token-decoding explainability.

Extractive summary: sentences quoted from the sources.