Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

Key points

• Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking.1
• Task over-specification causes spoilers, where solutions can bypass some of the harder steps the benchmark was supposed to measure.1
• Repository-level coding benchmarks have helped the field progress but experience unnecessary churn due to both over- and under-specification.1
• We show that leaking ground-truth information into the task statement causes over-specification in some of these benchmarks.1

Sources (1)

• [1]Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 12:02 AM

  Benchmarks drive AI progress, but are hampered by task under-specification and over-specification.

  Task under-specification causes guesswork, where solutions have to make things up, or worse, resort to contamination or reward hacking.

Extractive summary: sentences quoted from the sources.