Interaction-as-a-Judge

Key points

• Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games.1
• To this end, we collect and release two complementary datasets: $6,858$ pairs of applications with human preferences and corresponding agent traces, and $222$ applications with human-verified requirement labels.1
• Relative to standard non-interactive approaches, we find that Interaction-as-a-Judge improves evaluation in terms of predicting human preferences ($65.6% \to 69.1%$) and requirement verification ($77.0% \to 81.6%$).1
• Finally, through both inference- and training-time experiments, we demonstrate that Interaction-as-a-Judge can also improve the quality of generated artifacts.1

Sources (1)

• [1]Interaction-as-a-Judge
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 07:49 AM

  We study Interaction-as-a-Judge, an evaluation paradigm where computer-use agents (CUAs) directly interact with an application.

  Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games.

Extractive summary: sentences quoted from the sources.