Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments

Key points

• We find that they accept a correct value less often when their input holds no reference for it, that is, no text that says what the value should be.1
• The gap is not only a truncation artifact: correct default values have the largest gap on a benchmark with no truncation.1
• A controlled source-relocation test shows that judges follow the reference in their input: they favor a visible incorrect value over a correct value whose source is hidden.1
• We introduce DeRef (Derived Reference): a separate model reads the full history without seeing the candidate and writes a reference for the judge.1

Sources (2)

• [1]Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 02:52 PM

  Tool-call judges evaluate argument values proposed by LLM agents.

  We find that they accept a correct value less often when their input holds no reference for it, that is, no text that says what the value should be.

• [2]Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments
  Hugging Face Daily Papers · Oct 13, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.