When 1+1 Backfires: Rethinking Relevance-Diversity Balancing for Sample-Aware Visual Token Reduction

Key points

• Visual token compression has emerged as a critical technique for reducing the inference cost of Vision-Language Models.1
• Existing methods select tokens based on text relevance, visual diversity, or a fixed combination of both---treating the relevance-diversity balance as a universal constant independent of the input.1
• At the dataset level, grid-searching the optimal relevance-diversity weight per benchmark reveals substantial cross-task variation.1
• We first derive a Geometric Intent Prior (GIP) from the spatial concentration of text-relevant tokens to infer a sample-specific relevance-diversity weight, and then apply this weight to jointly select tokens via a Weighted Text Centroid (WTC) for relevance scoring and a Visual Marginal Coverage Gain (VMCG) for diversity.1

Sources (1)

• [1]When 1+1 Backfires: Rethinking Relevance-Diversity Balancing for Sample-Aware Visual Token Reduction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 11:31 AM

  Based on these observations, we propose SAVER, a framework for Sample-Aware relevance-diversity balanced Visual tokEn Reduction.

  Visual token compression has emerged as a critical technique for reducing the inference cost of Vision-Language Models.

Extractive summary: sentences quoted from the sources.