ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models

Key points

• Multimodal Large Language Models (MLLMs) encode images into dense visual token sequences, creating substantial attention and KV-cache overhead.1
• Existing visual token compression methods reduce this cost by pruning low-saliency tokens in shallow layers, but this practice couples temporary computational deactivation with irreversible information deletion.1
• Motivated by this finding, we propose ReTrace, a backtracking-aware compression framework that introduces a dormant token state.1
• Experiments on Qwen2.5-VL and InternVL3 demonstrate nearly 90% reductions in FLOPs and KV cache, a 2.83$\times$ prefilling speedup, and strong performance retention across complex vision-language benchmarks.1

Sources (1)

• [1]ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 04:19 AM

  We show that this coupling breaks an intrinsic inference pattern in MLLMs. Through layer-wise attention analysis, we identify Visual Backtracking: visual pivots remain spatially stable, yet in middle layers they actively retrieve context from tokens that appeared unimportant earlier.

  Multimodal Large Language Models (MLLMs) encode images into dense visual token sequences, creating substantial attention and KV-cache overhead.

Extractive summary: sentences quoted from the sources.