Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory

Key points

• Conventional surgical segmentation relies on closed-set categories or explicit text queries, which cannot resolve implicit queries that require reasoning about workflows, instrument functions, and temporal relationships.1
• Specifically, SurgReasonSeg constructs DT representations from surgical foundation models and trains two low-rank adapters (LoRA) of one large language model with reinforcement learning (RL), each specialized for a different query complexity.1
• SurgReasonSeg also sets the temporal context with an adaptive sliding window conditioned on query complexity and workflow phase, and retrieves relevant historical cases from a surgical context memory by workflow similarity and procedural relevance.1
• We further introduce EndoVis-ReasonSeg, a surgical reasoning segmentation benchmark with 1,460 training and 961 test samples across semantic, spatial, and temporal categories.1

Sources (1)

• [1]Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:10 PM

  To address these limitations, we propose SurgReasonSeg, a complexity-aware method that brings reasoning segmentation over digital twin (DT) representations to surgical video.

  Conventional surgical segmentation relies on closed-set categories or explicit text queries, which cannot resolve implicit queries that require reasoning about workflows, instrument functions, and temporal relationships.

Extractive summary: sentences quoted from the sources.