VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Key points

• Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data.1
• We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones.1
• Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones.1
• These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes.1

Sources (1)

• [1]VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
  Hugging Face Daily Papers · Oct 19, 12:00 AM

  We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites.

  Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data.

Extractive summary: sentences quoted from the sources.