SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking

Key points

• Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-In-First-Out rule, allowing temporal consistency in complex videos.1
• We instead present a more general, data-driven approach that frames memory control as a sequential decision-making task.1
• SAM3RL is an extension of the SAM 3 tracker with a lightweight memory controller, trained by reinforcement learning, that decides which frames enter the memory bank in order to maximize tracking performance.1
• SAM3RL sets a new state of the art on 10 of 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without any runtime overhead.1

Sources (1)

• [1]SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 10:17 AM

  Segment Anything Model 3 (SAM 3) is a foundation model for image and video segmentation that sets the state of the art in visual object tracking.

  Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-In-First-Out rule, allowing temporal consistency in complex videos.

Extractive summary: sentences quoted from the sources.