Beyond Calibration: Temporal LLM Confidence Assessment Across Languages and Local Settings

Key points

• Large language models are expected to estimate their own uncertainty, with confidence signals used to guide retrieval, adaptive compute allocation, routing, and selective abstention.1
• Inspired by classical meta-memory theory, we distinguish anticipatory pre-solve confidence (Feeling-of-Knowing, FOK) from retrospective post-solve confidence (Judgement-of-Learning, JOL).1
• We evaluate these signals across two frontier LLMs, multiple reasoning domains, translated Marathi benchmarks, and a newly constructed benchmark of genuinely local Marathi factual questions.1
• Confidence behaviour transfers surprisingly well under direct language translation, with calibration trends largely preserved between English and Marathi, but degrades substantially on local Marathi knowledge, suggesting that linguistic transfer alone is insufficient for region-specific information needs.1

Sources (1)

• [1]Beyond Calibration: Temporal LLM Confidence Assessment Across Languages and Local Settings
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:19 AM

  We investigate whether confidence signals measured at different temporal stages of problem solving encode distinct forms of epistemic information and remain reliable beyond English-centric settings.

  Large language models are expected to estimate their own uncertainty, with confidence signals used to guide retrieval, adaptive compute allocation, routing, and selective abstention.

Extractive summary: sentences quoted from the sources.