ATLAS: Learning to Adapt LVLM Attention for Reasoning Video Segmentation

Key points

• Although large vision-language models (LVLMs) implicitly encode localization cues in their attention maps, these attentions are often noisy, diffuse, and scattered across layers, heads, and modalities, limiting their direct use for segmentation.1
• We propose ATLAS, an attention adaptation framework that learns to transform raw LVLM attentions into object-centric localization cues.1
• A lightweight U-Net equipped with temporal attention and FiLM-based modulation fuses these complementary representations to generate coarse segmentation heatmaps, which are propagated across the video using SAM2.1
• Extensive experiments on challenging referring and reasoning video segmentation benchmarks show that ATLAS outperforms existing frozen-backbone methods.1

Sources (1)

• [1]ATLAS: Learning to Adapt LVLM Attention for Reasoning Video Segmentation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 05:03 PM

  Reasoning Video Object Segmentation (RVOS) aims to localize and segment objects referred to by natural-language expressions throughout a video.

  Although large vision-language models (LVLMs) implicitly encode localization cues in their attention maps, these attentions are often noisy, diffuse, and scattered across layers, heads, and modalities, limiting their direct use for segmentation.

Extractive summary: sentences quoted from the sources.