RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation

Key points

• Existing methods aggregate information across frames to exploit temporal context, but an emphasis on consistency can obscure these changes.1
• To address this issue, we propose Relation-Transition-Aware Transformer (RTAformer), a one-stage framework that separately models entity persistence and relation changes.1
• It injects calibrated differences between adjacent-frame relation queries to guide predicate updates, while cross-frame instance alignment during training helps reduce cross-frame representational drift.1
• Experiments on Action Genome show improvements in recall and mean recall over existing methods, with ablation studies supporting the contributions of the proposed components and comparisons using matched backbones showing larger gains on relation-transition frames than on non-transition frames.1

Sources (1)

• [1]RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 12:31 PM

  Video scene graph generation (VidSGG) aims to capture objects and their evolving relationships in videos.

  Existing methods aggregate information across frames to exploit temporal context, but an emphasis on consistency can obscure these changes.

Extractive summary: sentences quoted from the sources.