Bottlenecked semantic compression by representation reconstruction

Key points

• Vision Transformers incur high computational costs when processing long visual token sequences.1
• Existing reduction methods typically rank tokens independently and discard those deemed unimportant or redundant.1
• At each compression layer, a lightweight position-conditioned decoder reconstructs the current tokens, leaving shared information compressed in the SMB while explicitly preserving poorly reconstructed details.1
• Experiments on image classification and multimodal understanding demonstrate favorable performance-efficiency trade-offs: ReConV surpasses the uncompressed baseline with 48% fewer GMACs on ImageNet and enables reusable image-level representations for efficient multi-turn inference.1

Sources (1)

• [1]Bottlenecked semantic compression by representation reconstruction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 09:24 PM

  We propose a reconstruction-guided framework for visual token condensation (ReConV), which summarizes the complete visual sequence with a compact Summary Bank (SMB) and retains tokens whose information is not adequately captured by the SMB, as indicated by their reconstruction residuals.

  Vision Transformers incur high computational costs when processing long visual token sequences.

Extractive summary: sentences quoted from the sources.