A Theoretical Analysis of Environment-Aware Out-of-Distribution Detection for Vision-Language Models

Key points

• Continual vision-language models (VLMs) encounter a stream of environments and must also reject inputs that do not belong to the environment at hand.1
• We propose a cross-environment maximum-concept-matching (MCM) score with two terms: a primary term matching the test image against the current class texts, and a secondary term matching it against the class texts of a previous environment, weighted by the intra-modal alignment between the two environments.1
• We then show that the score gap of an MCM detector between two environments equals the self-contribution of the in-distribution environment minus its cross-modal contribution into the OOD environment, which separates soft OOD environments, with explicit AUROC, true-positive, and false-positive guarantees, from hard ones, which no threshold that retains the in-distribution data can reliably reject.1
• For cascaded detection, in which the text prototypes of all seen environments are pooled, the false-positive rate on every past environment is non-decreasing in time, and we characterize when detection provably fails.1

Sources (1)

• [1]A Theoretical Analysis of Environment-Aware Out-of-Distribution Detection for Vision-Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 07:43 PM

  We develop a theory of environment-aware OOD detection built on contribution scores, pairwise cosine expectations measuring how the image and text features of one environment align with those of another.

  Continual vision-language models (VLMs) encounter a stream of environments and must also reject inputs that do not belong to the environment at hand.

Extractive summary: sentences quoted from the sources.