Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates

Key points

• We pair premise-hypothesis items from $δ$-SNLI with one strengthener and one weakener each, elicit the prior and the posteriors in separate calls, and measure the directional movement s: the mean posterior change signed by the annotated direction, so that the prior cancels out.1
• We vary whether the update is stated ("Here is something else you learned") or reported ("Someone said that"), and whether the model gives one probability or five weighted estimates.1
• We test seven configurations: two local models (Qwen3-8B, OLMo-2-7B; 240 pairs) and five configurations run through agent command-line harnesses (120 pairs), with hypotheses and analysis rules fixed before the formal runs (not publicly registered).1
• Because $δ$-NLI annotates only direction, we make no claim about how large an update should be.1

Sources (1)

• [1]Hearsay or Fact? How Framing and Elicitation Shape LLM Probability Updates
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 07:05 AM

  Large language models (LLMs) are increasingly used as judgment assistants and as stand-in participants in psychology experiments, yet little is known about how much their probability estimates move in response to a single new piece of information.

  We pair premise-hypothesis items from $δ$-SNLI with one strengthener and one weakener each, elicit the prior and the posteriors in separate calls, and measure the directional movement s: the mean posterior change signed by the annotated direction, so that the prior cancels out.

Extractive summary: sentences quoted from the sources.