Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering

Key points

• We find that abstentions are lost at two stages: contrastive decoding with candidate filtering can exclude the abstention token before any comparison takes place, and the reference branch's response to the abstention instruction is removed when the reference prediction is subtracted.1
• We therefore propose instruction-preserving decoding, which keeps the effect of the abstention instruction on the original-image prediction intact while leaving the contrast strength freely adjustable.1
• Using a perturbed image and an image-free prediction as references, we obtain IP-VCD and IP-M3ID.1
• Candidate measurements and paired per-question comparisons show that computing the instruction effect and the visual contrast term separately lets contrastive decoding support both correct answers and supported abstentions.1

Sources (1)

• [1]Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answering
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 12:53 PM

  Visual contrastive decoding corrects answers using the prediction difference between the original image and a reference condition, yet the same contrast also turns supported abstentions into incorrect answers.

  We find that abstentions are lost at two stages: contrastive decoding with candidate filtering can exclude the abstention token before any comparison takes place, and the reference branch's response to the abstention instruction is removed when the reference prediction is subtracted.

Extractive summary: sentences quoted from the sources.