RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development

Key points

• Beyond basic tool use and coding skills, effectively incorporating visual references into agentic generation is equally essential for applying AI agents to real-world productivity tasks.1
• We study this crucial capability through webpage development, a practical and readily evaluable setting in which existing benchmarks remain largely idealized: they typically focus on reconstructing visual targets or generating webpages from textual instructions alone, overlooking the more common workflow of developing from visual examples.1
• We evaluate the resulting open-ended artifacts using independently verifiable keypoints and complementary programmatic and VLM judges.1
• Across nine frontier (M)LLMs, the strongest system achieves an overall score of only 52.12 out of 100, revealing that visual-reference-conditioned agentic generation remains a substantial challenge for current models in practical webpage-development settings.1

Sources (1)

• [1]RefCraft: Benchmarking Visual-Reference-Conditioned Agentic Generation through Web Development
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:55 PM

  Specifically, we introduce RefCraft, a benchmark comprising 300 realistic multimodal webpage-development tasks that combine natural-language instructions with screenshots from one to five reference webpages.

  Beyond basic tool use and coding skills, effectively incorporating visual references into agentic generation is equally essential for applying AI agents to real-world productivity tasks.

Extractive summary: sentences quoted from the sources.