Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

Key points

• Benchmarks shape how model capabilities are measured and interpreted, yet their scores can obscure whose conventions, contexts, and judgments define successful performance.1
• Across 16 resources, we find that American-centered conventions account for 62.4-94.4% of matched signals in 11 text benchmarks, while geographic visibility in five image-text resources is concentrated and strongly associated with national GDP share (mean Pearson r = 0.780; Spearman p = 0.778).1
• Documentation reveals a systematic source-target gap: all 16 resources operationalize source-user context, but none operationalize target-user context.1
• Paired scoring experiments further show that meaning-preserving answer variants reduce DROP exact-match accuracy from 100.0% to 33.3%, while TruthfulQA exhibits metric-dependent reductions in reference similarity.1

Sources (1)

• [1]Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 10:39 PM

  We introduce a user-centric audit framework that distinguishes users who contribute benchmark content from those whose needs evaluation results are intended to represent.

  Benchmarks shape how model capabilities are measured and interpreted, yet their scores can obscure whose conventions, contexts, and judgments define successful performance.

Extractive summary: sentences quoted from the sources.