browsecomp-plus-mini: 15 of BrowseComp-Plus's 830 queries that rank deep-research agents roughly like the full set, no LLM judge

Key points

• So I built a 15-query subset of the full 830 that ranks systems roughly the way the full benchmark does.1
• weighted Spearman rank agreement 0.88 averaged over 30 splits, versus 0.75 for a random 151
• two runs more than 10 points apart on the full benchmark keep their order about 96% of the time; 5 to 10 points apart, about 75%1
• all answers are 1-4 words, so it grades by normalized exact match, no LLM judge1

Sources (1)

• [1]browsecomp-plus-mini: 15 of BrowseComp-Plus's 830 queries that rank deep-research agents roughly like the full set, no LLM judge
  r/LocalLLaMA (top, daily) · Oct 18, 02:30 PM

  I needed a cheap way to compare search agents on BrowseComp-Plus without burning hours of agent time every run.

  So I built a 15-query subset of the full 830 that ranks systems roughly the way the full benchmark does.

Extractive summary: sentences quoted from the sources.