VHA: VQA-based Hallucination Assessment for Generative Image Super-Resolution

Key points

• Generative image super-resolution produces visually realistic high-resolution images, but may also introduce hallucinations by adding unsupported content or altering existing details.1
• The proposed approach uses a vision-language model (VLM) to determine whether the semantic content and low-level visual evidence in a super-resolved image are supported by its ground-truth counterpart.1
• We therefore develop VHA+, a learned, lightweight model that directly predicts VHA scores from ground-truth and super-resolved image pairs.1
• We construct a hallucination benchmark, VHA-bench, through a human study and show that both VHA and VHA+ correlate highly with human judgments of hallucination.1

Sources (1)

• [1]VHA: VQA-based Hallucination Assessment for Generative Image Super-Resolution
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 04:34 AM

  Because existing super-resolution metrics fail to measure whether generated details remain grounded in the reference image, we present a hallucination assessment technique based on bidirectional visual question answering (VQA), which we term VQA-based Hallucination Assessment (VHA).

  Generative image super-resolution produces visually realistic high-resolution images, but may also introduce hallucinations by adding unsupported content or altering existing details.

Extractive summary: sentences quoted from the sources.