PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA

Key points

• Text-to-video (T2V) generation models can produce visually realistic videos that nevertheless violate the physical consequences implied by the prompt and scene.1
• PhyMetric contains 1{,}115 scenes spanning 50 physical categories across six domains, paired with 8{,}920 answer-balanced binary questions over 12 evaluation dimensions.1
• At evaluation time, a vision-language model (VLM) judge answers each question from question-aware sampled frames conditioned on the prior, and correctness is hierarchically aggregated into scene, dimension, physical category, and model scores.1
• Experiments on four open-source T2V models and three VLM judges show that scene-specific binary QA is a critical component of the protocol: when each targeted question is replaced by a single generic plausibility query, evaluator accuracy drops to the chance level.1

Sources (1)

• [1]PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 09:00 AM

  We introduce PhyMetric, a scene-level diagnostic benchmark that recasts physical plausibility evaluation as verifiable, scene-grounded question answering.

  Text-to-video (T2V) generation models can produce visually realistic videos that nevertheless violate the physical consequences implied by the prompt and scene.

Extractive summary: sentences quoted from the sources.