Can Synthetic UAV Benchmarks Be Trusted? Real-Synthetic Validation for UAV Model Selection

Key points

• We study when Synthetic evaluation preserves the model-selection conclusions obtained from Real UAV data through two complementary cases under fixed evaluation protocols.1
• In a requirement-specific road-debris benchmark, four language-conditioned perception systems obtain lower absolute F1 on Synthetic data, yet the point-estimate ordering and top-ranked model are preserved; bootstrap analysis yields no supported pairwise reversal, and size-matched resampling preserves the top-model decision.1
• A complementary audit of five UAV business scenarios further reveals substantial Real-Synthetic mismatch in class composition, object density, and target scale, including systematic under-representation of small targets.1
• These results show that Synthetic UAV benchmark trust is conditional: semantic coverage is insufficient, and validity must be established empirically for the task, operating regime, and model panel of interest.1

Sources (1)

• [1]Can Synthetic UAV Benchmarks Be Trusted? Real-Synthetic Validation for UAV Model Selection
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 06:57 AM

  Synthetic UAV imagery can expand pre-deployment test evidence when task-specific Real data are scarce, but visual plausibility or semantic label alignment alone does not establish validity for model selection.

  We study when Synthetic evaluation preserves the model-selection conclusions obtained from Real UAV data through two complementary cases under fixed evaluation protocols.

Extractive summary: sentences quoted from the sources.