A Measurement Perspective on Benchmarking LLM-Generated Text Detectors using Item Response Theory
We propose a measurement-based benchmarking framework for jointly characterizing benchmark heterogeneity and detector behavior using multidimensional item response theory (IRT).
ProofPaper ↗
Key points
- Benchmarking large language model (LLM)-generated text detectors commonly relies on aggregate detector-level metrics such as accuracy, F1 score, and AUC.
- By treating benchmark texts as measurement items and detectors as respondents, the framework estimates text-level discrimination and difficulty-location parameters alongside multidimensional detector abilities.
- We apply the framework to a benchmark of 32K human-written and machine-generated texts across four domains, evaluated by 14 detectors.
- Detector ability profiles reveal multidimensional and domain-dependent response patterns that are not fully reflected in aggregate performance rankings.
Sources (1)
- [1]A Measurement Perspective on Benchmarking LLM-Generated Text Detectors using Item Response TheoryarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 03:06 AM
We propose a measurement-based benchmarking framework for jointly characterizing benchmark heterogeneity and detector behavior using multidimensional item response theory (IRT).
Benchmarking large language model (LLM)-generated text detectors commonly relies on aggregate detector-level metrics such as accuracy, F1 score, and AUC.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 18, 2026ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference
- Oct 17, 2026Pair-Aware Optimization: Coordinating Exploration and Refinement for Jailbreaking LLM Agents
- Oct 17, 2026TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference
- Oct 16, 2026Deep LLM layers expose passage-level context in EEG during silent reading
- Oct 16, 2026OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners
- Oct 15, 2026AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents
