Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs

Key points

• Multimodal large language models have advanced rapidly, but most remain optimized for single-image inputs.1
• Multi-image reasoning is costly because each image adds many visual tokens, increasing encoding and attention demands.1
• In this work, we study scalable multi-image reasoning and understanding without retraining the base model and without accessing attention matrices at inference time.1
• To this end, we propose mmVB, a lightweight plug-and-play module that assigns query-conditioned per-image token budgets via cross-image context, adding only 1.6M trainable parameters while reducing visual input tokens by 9x.1

Sources (1)

• [1]Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:59 PM

  Multi-image reasoning introduces additional challenges: redundancy may occur across images, task-relevant evidence may be sparse or distributed, and aggressive per-image compression can remove details needed for comparison, counting, temporal reasoning, retrieval, or evidence grounding.

  Multimodal large language models have advanced rapidly, but most remain optimized for single-image inputs.

Extractive summary: sentences quoted from the sources.