SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

Key points

• Recent agents have demonstrated strong capabilities in generating images from textual prompts, motivating their utilization for scientific diagram generation.1
• Scientific diagrams naturally span diverse types, each characterized by distinct visual structures and content organization.1
• However, existing VLM-based evaluation benchmarks fail to reflect the above human evaluation practices for two main reasons: 1) benchmark datasets typically lack a systematic categorization of different diagram types and thus adopt the same generic criteria for all diagrams, and 2) their evaluation criteria are often designed heuristically, without explicit alignment to human evaluation preferences.1
• Based on this dataset, SciDiagBench develops an evaluation method that leverages a preference-guided calibration process to incorporate human preferences into the generation of type-specific criteria.1

Sources (1)

• [1]SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 08:27 AM

  Motivated by these limitations, we introduce SciDiagBench, which consists of a benchmark dataset that systematically categorizes scientific diagrams into 11 types and a human preference-guided evaluation method.

  Recent agents have demonstrated strong capabilities in generating images from textual prompts, motivating their utilization for scientific diagram generation.

Extractive summary: sentences quoted from the sources.