ResearchResearch paperEfficiency & Inference · Large Language Models · Training & Scaling1 source · Oct 17, 2026

Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count

Visual token pruning aims to reduce inference cost while preserving the capabilities of the original VLM.

We reveal two limitations: 1) Aggregate accuracy can obscure whether a pruning method truly identifies critical visual tokens; many examples remain solvable under arbitrary token removal and models can exploit textual shortcuts when visual evidence is degraded.
We introduce PAA-Bench, a plug-and-play evaluation framework for existing benchmarks, to evaluate whether pruning succeeds through meaningful token selection and whether this success translates into real speedup.
To determine when token choice actually matters, we propose Random-Pass@k to measure how often an example remains correct under random pruning, enabling comprehensive evaluation across different levels of selection difficulty.
To compare efficiency fairly, we introduce Preservation-Constrained Speedup (PCS) to report the maximum end-to-end speedup each method achieves under a shared preservation target.

[1]Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count
arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 03:15 PM
Visual token pruning aims to reduce inference cost while preserving the capabilities of the original VLM.
We reveal two limitations: 1) Aggregate accuracy can obscure whether a pruning method truly identifies critical visual tokens; many examples remain solvable under arbitrary token removal and models can exploit textual shortcuts when visual evidence is degraded.

Extractive summary: sentences quoted from the sources.

Before this