Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation

Key points

• We propose FLAIR, a Flow-guided LatentAction Injection framework for Reference-free surgical video generation.1
• FLAIR learns action priors from optical flow of real surgical videos, dynamically predicts corresponding latent action representation from an input prompt, and injects it into a frozen base model to generate surgical videos with improved action consistency.1
• We further construct SurgActionClip-30K, the first large-scale surgical vision dataset comprising action-centric segmented clips and structured caption labels, addressing the persistent lack of fine-grained, action-centric surgical datasets.1
• Lastly, we introduce SurgMetrics, the first surgical domain-specific evaluation metrics for quantifying the quality of generated surgical videos, addressing the persistent absence of clinically grounded evaluation standards in this domain.1

Sources (1)

• [1]Beyond Masks and Trajectories: Flow-Guided Latent Action Injection for Stable Surgical Video Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 7, 10:14 AM

  Surgical video generation holds substantial potential for surgical education, simulation, and data augmentation, yet generating surgical videos with realistic and clinically plausible motion remains challenging.

  We propose FLAIR, a Flow-guided LatentAction Injection framework for Reference-free surgical video generation.

Extractive summary: sentences quoted from the sources.