OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training

Key points

• We identify a temporal generation gap: the internal attention of the model localizes the target moment more accurately than the timestamps it generates.1
• To close this gap, we propose TACG (Temporal Attention-Consistency Grounding), a post-training method that treats internal attention as a reliable temporal pivot and aligns it with both the model prediction and the ground-truth annotation through attention-consistency training.1
• We further build a 580K-sample training dataset by reconstructing temporal QA from dense audio-visual annotations and organizing general omni-modal instruction data.1
• Using TACG and this training dataset, we train OmniGround, a general-purpose omni-modal model with stronger temporal grounding capability, without injecting any additional temporal cues.1

Sources (1)

• [1]OmniGround: Strengthening Omni-Modal Temporal Grounding via Attention-Consistency Training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 07:27 AM

  Omni-modal large language models can jointly understand video, audio, and text, yet they remain weak at temporal grounding, a capability essential for real-world audio-visual applications.

  We identify a temporal generation gap: the internal attention of the model localizes the target moment more accurately than the timestamps it generates.

Extractive summary: sentences quoted from the sources.