UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

Key points

• First, due to the significant discrepancy in information density, video requires prolonged high noise to organize spatial and motion structure, whereas the sparser audio signal can settle earlier during the synthesis process, the dynamics a uniform schedule might compromise.1
• Second, lockstep denoising leaves asymmetric workflows, such as conditioning on one modality to synthesize or edit another, untrained.1
• We present UniAVC, which addresses both obstacles.1
• Modality-decoupled schedulers meanwhile warp the noise schedule independently from a shared timestep, while subtasks with one clean modality directly supervise asymmetric configurations.1

Sources (2)

• [1]UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling
  Hugging Face Daily Papers · Oct 17, 12:00 AM

  Unified audio-video creation requires a single model capable of text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing across both modalities.

  First, due to the significant discrepancy in information density, video requires prolonged high noise to organize spatial and motion structure, whereas the sparser audio signal can settle earlier during the synthesis process, the dynamics a uniform schedule might compromise.

• [2]UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 05:46 PM · same content

Extractive summary: sentences quoted from the sources.