Edit in Sync: Unified End-to-End Audio-Visual Editing for Human-Centric Videos

Key points

• Existing methods process inputs through independent video and audio pipelines, preventing cross-modal interaction between visual editing and audio dubbing.1
• We present AVE, the first unified end-to-end framework that jointly performs reference-guided person swapping, insertion, and deletion across both audio and video modalities without requiring spatial masks.1
• To overcome data scarcity, we design a synthesis pipeline and introduce AVE-Dataset, the first audio-visual editing dataset to provide 100K pairs with appearance and voice references.1
• AVE outperforms modality-separate audio-visual editing pipelines in editing realism, temporal consistency, and audio-visual alignment.1

Sources (1)

• [1]Edit in Sync: Unified End-to-End Audio-Visual Editing for Human-Centric Videos
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 09:53 AM

  Human-centric audio-visual editing requires precise alignment of lip movements, gestures, and expressions with speech.

  Existing methods process inputs through independent video and audio pipelines, preventing cross-modal interaction between visual editing and audio dubbing.

Extractive summary: sentences quoted from the sources.