VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

Key points

• Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages.1
• We evaluate leading single-vector and multi-vector visual retrievers together with diagnostic retrieval settings such as document-context scoring, query splitting, reranking, and round-robin ensembles.1
• The strongest evaluated configuration reaches 69.3 nDCGAll@10 and 80.6 RecallAll@10 under structure-balanced Macro evaluation, but source- and evidence-structure breakdowns show that cross-document retrieval remains substantially harder than single-page retrieval, and that document-context scoring is especially helpful for within-document multi-page queries.1
• VL-DocIR exposes these failure modes and provides a more demanding target for visual long-document retrieval.1

Sources (1)

• [1]VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 04:25 PM

  We present VL-DocIR, a page-level benchmark for evidence-complete visual long-document retrieval.

  Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages.

Extractive summary: sentences quoted from the sources.