SAPO: Self-Supervised Attention Policy Optimization for Improving Temporal Video Grounding in Vision-Language Models

Key points

• Existing gains from explicitly aligning video frames with absolute time suggest that VLMs may already encode accurate temporal locations internally but lose precision when expressing them as timestamps.1
• Because precise temporal annotations are expensive, this finding motivates closing the gap to improve temporal grounding using only unlabeled video-query pairs.1
• We propose Self-Supervised Attention Policy Optimization (SAPO), which uses internal temporal attention as self-supervision for Group Relative Policy Optimization and applies a ground-truth-free criterion to select high-value samples.1
• SAPO consistently improves temporal grounding across multiple datasets and models spanning weak to strong initial grounding ability.1

Sources (1)

• [1]SAPO: Self-Supervised Attention Policy Optimization for Improving Temporal Video Grounding in Vision-Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 07:23 AM

  Video vision-language models (VLMs) have strong semantic understanding, yet remain unreliable at precise temporal grounding.

  Existing gains from explicitly aligning video frames with absolute time suggest that VLMs may already encode accurate temporal locations internally but lose precision when expressing them as timestamps.

Extractive summary: sentences quoted from the sources.