AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

Key points

• As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations.1
• We instantiate the metric as AVC-Reward, a trainable audio-video reward model.1
• To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models.1
• AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal.1

Sources (2)

• [1]AVC-Reward: Event-Centric Audio-Video Consistency Evaluation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:51 PM

  We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric.

  As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations.

• [2]AVC-Reward: Event-Centric Audio-Video Consistency Evaluation
  Hugging Face Daily Papers · Oct 15, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.