Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models

Key points

• Prior work usually measures this through one channel: verbalized confidence, a linear probe on hidden states, or agreement among sampled answers (semantic entropy), and often treats these as interchangeable.1
• We compare all three in a pre-registered audit of five Qwen2.5 models (0.5B-7B, including a base/instruct pair at 7B) on factual retrieval (PopQA, SimpleQA) and multi-step reasoning (GSM8K, MATH).1
• First, probes trained on prefill states to predict whether sampled answers will disagree become informative earlier for retrieval (about 7% of layer depth on average) than for reasoning (about 28%).1
• Third, Expected Calibration Error is misleading: verbal confidence looks well calibrated because it clusters near the base rate while discriminating poorly, whereas sampling agreement has worse ECE but about twice the resolution.1

Sources (1)

• [1]Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 06:04 AM

  Whether a language model "knows what it knows" matters for trustworthy deployment and for deciding when it should abstain.

  Prior work usually measures this through one channel: verbalized confidence, a linear probe on hidden states, or agreement among sampled answers (semantic entropy), and often treats these as interchangeable.

Extractive summary: sentences quoted from the sources.