SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition

Key points

• The assessment of surgical skills in robotic surgery training has traditionally relied on subjective evaluations by human experts, creating bottlenecks in surgical education and limiting scalability.1
• Combining a visual encoder with a transformer architecture, the model performs three key tasks simultaneously: 1) surgical task classification (knot tying, needle passing, suturing), 2) skill level assessment on a standardized scale, and 3) natural language generation of surgical gesture descriptions.1
• The model processes video sequences from the da Vinci surgical robot and generates human-readable text descriptions of surgical actions, similar to how language models generate text from prompts, using a multi-task learning framework with adaptive uncertainty weighting to optimize performance across all objectives.1
• Evaluated on the da Vinci Skills Simulator dataset, the proposed framework demonstrates state-of-the-art performance in surgical task recognition and provides interpretable skill assessments that correlate with expert evaluations, with the model's ability to generate natural language descriptions of surgical gestures opening new possibilities for automated surgical education, real-time feedback systems, and objective skill assessment in surgical training and certification programs.1

Sources (1)

• [1]SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 05:33 PM

  We present a novel foundation model that automatically analyzes robotic surgical videos to provide comprehensive skill assessment and gesture understanding.

  The assessment of surgical skills in robotic surgery training has traditionally relied on subjective evaluations by human experts, creating bottlenecks in surgical education and limiting scalability.

Extractive summary: sentences quoted from the sources.