Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

Key points

• Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer.1
• Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a video reference gap between linguistic inference and the evidence distributed across frames.1
• We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning.1
• Extensive experiments across five VRS benchmarks and three model backbones demonstrate the strong performance of TCoVP, highlighting the effectiveness of organizing spatial and temporal evidence within the reasoning process.1

Sources (2)

• [1]Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation
  Hugging Face Daily Papers · Oct 18, 12:00 AM

  We introduce Temporal Chain of Video Primitives (TCoVP), a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain.

  Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer.

• [2]Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 07:54 AM · same content

Extractive summary: sentences quoted from the sources.