Reasoning-Aware Multicalibrated Reliability Predictors for Language Models

Key points

• As people increasingly rely on large language models (LLMs) for information, model providers need to communicate meaningful estimates of whether individual responses are correct or, more generally, reliable.1
• In the context of hallucinations, existing work uses multicalibration to make such reliability predictions meaningful across application-relevant groups, such as prompt topics.1
• Empirically, we apply our framework to hallucination prediction and unsafe-response detection across a variety of benchmarks, and find that adding CoT groups to topic-based calibration reduces Brier score in both tasks.1
• Interestingly, we find that multicalibrating with respect to CoT groups alone (without any topic information) already improves calibration across topics, suggesting that the predictor's errors depend on how the model reasoned, rather than what it was asked about in the prompt.1

Sources (1)

• [1]Reasoning-Aware Multicalibrated Reliability Predictors for Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 04:06 PM

  In this work, we introduce reasoning-aware multicalibration, a framework that supplements application-relevant topic groups with groups derived from observable properties of CoT traces.

  As people increasingly rely on large language models (LLMs) for information, model providers need to communicate meaningful estimates of whether individual responses are correct or, more generally, reliable.

Extractive summary: sentences quoted from the sources.