Real PMIDs, Wrong Papers: Harness Design for Biomedical Claim-Verification Agents

Key points

• We run a controlled ablation on biomedical claim verification (SciFact) with the live PubMed API as the only tool.1
• Holding the backbone and output format fixed, we compare four harnesses: closed-book answering, single-shot retrieval using the claim as the query, a ReAct-style agent that writes its own PubMed queries, and an oracle given the gold abstracts, each with two efficient frontier backbones.1
• Naive single-shot retrieval finds the gold paper for only 6% of claims and turns overconfident errors into abstentions without improving accuracy.1
• Finally, a citation audit shows that closed-book models cite PubMed IDs that almost always exist (327/335) but are almost never relevant: a cross-family LLM judge rates only 3 of 327 existing citations as relevant to the claim, versus 67% for citations produced by the search agent.1

Sources (1)

• [1]Real PMIDs, Wrong Papers: Harness Design for Biomedical Claim-Verification Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 02:35 PM

  Literature-searching language agents are becoming a standard component of scientific assistants for the life sciences, yet there is little controlled evidence on which parts of their harness matter.

  We run a controlled ablation on biomedical claim verification (SciFact) with the live PubMed API as the only tool.

Extractive summary: sentences quoted from the sources.