HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference

Key points

• Evaluating software agents on a full benchmark is expensive.1
• A small task subset can preserve agent rankings while yielding inaccurate scores.1
• Observed target outcomes update the posteriors over the target latent state and group rates; posterior-guided acquisition selects subsequent tasks.1
• Observed successes and posterior-predictive counts for unqueried tasks yield a full-score posterior.1

Sources (2)

• [1]HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference
  Hugging Face Daily Papers · Oct 17, 12:00 AM

  We introduce Hierarchical Score Posterior Recovery (HSPR), which combines source-fitted response factors with uncertain group success rates.

  Evaluating software agents on a full benchmark is expensive.

• [2]HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 10:32 AM · same content

Extractive summary: sentences quoted from the sources.