Hierarchical and Query-Aware Visual Token Pruning for High-Resolution LVLMs

Key points

• However, the resulting thousands of vision tokens introduce severe computational bottlenecks during Large Language Model (LLM) inference.1
• Existing token pruning methods rely primarily on query-agnostic visual saliency, while current query-aware approaches either operate at a coarse crop-level resolution or compute token relevance too late, after vision tokens have already entered the heavy LLM layers.1
• To address this, we propose a training-free, hierarchical token pruning framework that utilizes query guidance prior to executing any LLM blocks.1
• Finally, at the token-level, we exploit projected vision tokens alongside textual query token embeddings in the shared space to measure fine-grained alignment, applying a text-debiasing normalization to produce high-fidelity query scores.1

Sources (1)

• [1]Hierarchical and Query-Aware Visual Token Pruning for High-Resolution LVLMs
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 06:32 PM

  High-resolution Large Vision-Language Models (LVLMs) preserve critical, fine-grained visual details by encoding a global thumbnail alongside multiple local crops.

  However, the resulting thousands of vision tokens introduce severe computational bottlenecks during Large Language Model (LLM) inference.

Extractive summary: sentences quoted from the sources.