Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching

Key points

• Vision--language pre-training has enabled substantial progress in image--text retrieval and zero-shot recognition by learning transferable cross-modal representations.1
• Global Compatibility Calibration (GCC) learns a sentence-level similarity metric, while Image-Conditioned Evidence Refinement (ICER) incorporates visual patch context into token queries and uses the resulting change in local matching evidence to adjust the global score.1
• We further introduce Cross-Image Margin Ranking (CIMR), which compares the same positive--negative caption pair across matched and mismatched images, encouraging a larger ranking margin when the image supports the correct description.1
• Experiments on compositional matching and attribute--object binding benchmarks show that Flow-CLIP improves binding accuracy over pretrained CLIP and learned global alignment through local evidence refinement, while cross-image ranking supervision mitigates reliance on language priors.1

Sources (1)

• [1]Flow-CLIP: Image-Conditioned Residual Scoring for CompositionalImage-Text Matching
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 02:12 PM

  To address this limitation, we propose Flow-CLIP, which refines global image-text alignment through image-conditioned changes in token--patch evidence.

  Vision--language pre-training has enabled substantial progress in image--text retrieval and zero-shot recognition by learning transferable cross-modal representations.

Extractive summary: sentences quoted from the sources.