LiveSync: Iterative Self-Training for Lip-Sync Generation
Audio-driven lip synchronization aims to match target speech through localized mouth and jaw articulation while preserving subject identity, head pose, and surrounding facial context.
ProofPaper ↗
Key points
- Naturally recorded videos provide synchronized reconstruction targets but rarely offer pairs that are matched in identity and pose yet differ primarily in speech-related articulation.
- Mask-based training enables reconstruction from these videos, but random-reference inputs differ from the aligned self-reference used at inference.
- We present LiveSync, an iterative self-training framework in which a single mask-based model generates pseudo pairs and is refined for deployment.
- Experimental results show that LiveSync achieves leading synchronization performance on HDTF and competitive performance compared with the baselines on the AIGC-LipSync subset, while maintaining low inference latency.
Sources (1)
- [1]LiveSync: Iterative Self-Training for Lip-Sync GenerationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 02:46 PM
Audio-driven lip synchronization aims to match target speech through localized mouth and jaw articulation while preserving subject identity, head pose, and surrounding facial context.
Naturally recorded videos provide synchronized reconstruction targets but rarely offer pairs that are matched in identity and pose yet differ primarily in speech-related articulation.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026Momentum: Persistent Dynamics for Video World Models
- Oct 16, 2026Persistent Planar Memory for Video World Models
- Oct 15, 2026NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning
- Oct 15, 2026The Density of Thought: Predicting and Steering Reasoning in Large Language Models
- Oct 15, 2026SUPRA-TEX: 3D Generation with Super-Aligned Texture
- Oct 14, 2026Two-Stage Score Estimation for Group-Specific Diffusion Models