When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation
Recent trackers built on foundation models such as the Segment Anything Model (SAM) achieve strong performance by propagating object masks through memory.
ProofPaper ↗
Key points
- However, this strength is also a key source of failure: under occlusion, reappearance, or distractors, an incorrect prediction can enter the tracking loop and be propagated over time, resulting in persistent identity drift.
- We address this gap with Dual-Agent, a failure-aware tracking framework that complements propagation with explicit verification and recovery.
- A vision-language verification agent first reasons over spatio-temporal context to determine whether the tracker is still following the target.
- Our results show that explicitly reasoning about when tracking fails and how to recover provides a complementary path to robust tracking beyond improving propagation alone.
Sources (1)
- [1]When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object SegmentationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 09:24 PM
Recent trackers built on foundation models such as the Segment Anything Model (SAM) achieve strong performance by propagating object masks through memory.
However, this strength is also a key source of failure: under occlusion, reappearance, or distractors, an incorrect prediction can enter the tracking loop and be propagated over time, resulting in persistent identity drift.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 11, 2026SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
- Oct 10, 2026We Have Been Interpreting Diffusion Transformer Backbone Behaviors Wrong
- Oct 9, 2026RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulation
- Oct 9, 2026Beyond Actions: Learning Future Operation Targets for Vision-Language-Action Models
- Oct 9, 2026Latent World Action Model for GUI Agents
- Oct 9, 2026Jag: Direct Box Prediction for Efficient Visual Grounding