Mean Accuracy Is the Wrong Estimand for a Release Decision
Auditing $2\,124$ fine-tuned models across eleven language models, we show that shipping models based on mean accuracy violates a $10%$ deployment failure budget $48.0%$ of the time.
ProofPaper ↗
Key points
- A release decision about a fine-tuned model is an operational commitment about the next rerun of its training recipe.
- These target fundamentally different questions: release decisions depend on catastrophic tail risk, whereas standard evaluation reports only central tendency.
- Common heuristics like $\bar b - 2σ$ prove equally dangerous: by underestimating variance on small samples, they become most permissive when evidence is weakest.
- We introduce recipe cards, an evaluation framework that directly certifies rerun tail failure rates using exact finite-sample binomial bounds.
Sources (1)
- [1]Mean Accuracy Is the Wrong Estimand for a Release DecisionarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 02:08 PM
Auditing $2\,124$ fine-tuned models across eleven language models, we show that shipping models based on mean accuracy violates a $10%$ deployment failure budget $48.0%$ of the time.
A release decision about a fine-tuned model is an operational commitment about the next rerun of its training recipe.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026LIBERO-Industry: Benchmarking Robotic Manipulation under Industrial Process Constraints
- Oct 12, 2026G$^2$WAM: World Action Models for Efficient and Geometry Grounded Image-Goal Navigation
- Oct 12, 2026A Survey of Flow Matching in Reinforcement Learning
- Oct 11, 2026$Δ$ Flow Matching for Robot Foundation Models
- Oct 10, 2026[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning
- Oct 10, 2026Co-adaptive Rule Routing and Constraining for Structured Knowledge Retrieval