GARRET: Generative Feedback for Region Retrieval in Visual Documents

Key points

• Multimodal retrieval-augmented generation over visually rich documents requires evidence at a finer granularity than entire pages.1
• A shared context-preserving retriever produces page and region multi-vector representations.1
• An auxiliary vision-language model attends to each crop separately, aggregates crop-specific contexts using the routing weights, and predicts the question followed by the answer under a next-token prediction objective.1
• We introduce ViDoRe-V3-Region, a corpus-wide benchmark containing 200,682 fixed candidate regions across eight domains.1

Sources (1)

• [1]GARRET: Generative Feedback for Region Retrieval in Visual Documents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 08:19 AM

  We introduce GARRET, a region-retrieval framework trained from (page, question, answer) triplets through differentiable generative feedback.

  Multimodal retrieval-augmented generation over visually rich documents requires evidence at a finer granularity than entire pages.

Extractive summary: sentences quoted from the sources.