Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

Key points

• While recent work leverages Large Language Models (LLMs) as knowledge sources, the question of how deeply such guidance should be integrated into a vision language architecture remains underexplored.1
• In this work, we generate structured reasoning guidance, visual focus concepts, distractor elements and external knowledge using Llama 3.3 70B and systematically compare two integration strategies within a unified frozen ViT + FLAN-T5 framework on the OK-VQA benchmark: decision level late fusion and attention-matrix injection.1
• Contrary to the prevailing assumption that deeper integration is superior, we find that shallow late fusion achieves the strongest results (39.4% exact match, +10.0 over the no guidance baseline), outperforming deep attention modification (31.8%) under data scarce fine-tuning.1
• We further contribute an LLM based augmentation pipeline that expands training data more than tenfold via question paraphrasing.1

Sources (2)

• [1]Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 04:45 PM

  Knowledge-based Visual Question Answering (VQA) requires reasoning that extends beyond visible image content, demanding external commonsense and factual knowledge.

  While recent work leverages Large Language Models (LLMs) as knowledge sources, the question of how deeply such guidance should be integrated into a vision language architecture remains underexplored.

• [2]Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?
  Hugging Face Daily Papers · Oct 12, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.