ABACUS: A Benchmark for Image Editing Models to Assess Object Category and Counting Under Scene Conservation

Key points

• Current benchmarks largely miss this problem: they score edits with holistic similarity metrics or with vision-language model (VLM) judges that are themselves poor at counting, so numerical errors go unnoticed or are graded incorrectly.1
• As a result, the true numerical ability of today's editing models has never been accurately measured.1
• We address this issue with ABACUS, a new benchmark, evaluation protocol and dedicated scoring metric for numerical image editing.1
• By evaluating leading image editors, our method reveals substantial numerical editing failures that worsen with increasing scene complexity and object count; failures that prior benchmarks often overlooked or misjudged.1

Sources (1)

• [1]ABACUS: A Benchmark for Image Editing Models to Assess Object Category and Counting Under Scene Conservation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 02:53 PM

  Instruction-based image editing models can now add, remove a specified number of objects or change their category while preserving the scene.

  Current benchmarks largely miss this problem: they score edits with holistic similarity metrics or with vision-language model (VLM) judges that are themselves poor at counting, so numerical errors go unnoticed or are graded incorrectly.

Extractive summary: sentences quoted from the sources.