Effectiveness of Hidden-State Probes for Localizing Reasoning Failures
We test the hidden states of 4B and 8B reasoning models for the failure type and the failing step and compare each probe with the strongest baseline that uses no hidden states.
ProofPaper ↗
Key points
- Verifying the final answer of a long reasoning trace may show that the trace is wrong but it does not distinguish three cases.
- Standard hidden-state probes are trained on final-answer correctness, a label that conflates these cases.
- Monte Carlo rollouts distinguish the cases and locate the failing step at additional inference cost.
- On incorrect 8B traces, the confidence probe, a calibrated correctness probe at a midpoint cutoff, misses 81% of cases in which a single step makes the trace unrecoverable and 47% of cases in which another attempt would succeed.
Sources (1)
- [1]Effectiveness of Hidden-State Probes for Localizing Reasoning FailuresarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 03:15 PM
We test the hidden states of 4B and 8B reasoning models for the failure type and the failing step and compare each probe with the strongest baseline that uses no hidden states.
Verifying the final answer of a long reasoning trace may show that the trace is wrong but it does not distinguish three cases.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
- Oct 15, 2026nvidia/Nemotron-3.5-Rerank-8B-BF16
- Oct 15, 2026Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling
- Oct 14, 2026ibm-granite/granite-switch-4.1-30b
- Oct 14, 2026The Design Space of Tri-Modal Masked Diffusion Models
- Oct 14, 2026CoTranslator: Reverse to Forward Chain-of-Thought Translator for Reasoning Distillation