DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM Systems
Decision models support routing, guardrails, and output verification in LLM systems.
ProofPaper ↗
Key points
- However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions.
- We introduce DecisionPivot, a diagnostic benchmark comprising 48 tasks and 40,033 items across seven decision roles.
- We evaluate 40 decision models and 14 general models using documented, model-specific protocols, assessing item and contrast-group accuracy, probability ordering, calibration, and empirical selective coverage.
- By linking errors to pivotal details and difficulty levels, DecisionPivot provides fine-grained diagnostics of decision failures that aggregate evaluation alone cannot reveal.
Sources (2)
- [1]DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM SystemsHugging Face Daily Papers · Oct 20, 12:00 AM
Decision models support routing, guardrails, and output verification in LLM systems.
However, item accuracy alone cannot establish whether models make correct decisions across controlled variations or assign reliable confidence to their predictions.
- [2]DecisionPivot: A Diagnostic Benchmark for Decision Models in LLM SystemsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:57 PM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models
- Oct 19, 2026Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
- Oct 18, 2026vllm-project/vllm v0.32.0
- Oct 18, 2026Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark: DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw on 2x NVIDIA DGX Spark (GB10) with TensorFold: 2.1-2.3x the vLLM kit's decode speed.
- Oct 18, 2026CoEvo-RSI: When Harnesses Evolve, Which Models Should Follow?
- Oct 18, 2026ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference