Is Sycophancy a Form of In-Context Updating?
We argue that this framing over-emphasizes the user and propose viewing sycophancy as a failure within the broader and generally useful process of in-context updating, by which models revise their answers based on information supplied during an interaction.
ProofPaper ↗
Key points
- Factual sycophancy, a language model's deference to a user's false beliefs, is usually understood as a bid for the user's approval at the expense of truthfulness.
- We support this view with behavioral and mechanistic evidence.
- First, models' responses to user claims sit within a broader pattern of source-sensitive updating.
- Our results suggest that factual sycophancy arises when in-context updating gives a user's assertion excessive weight relative to its evidence.
Sources (1)
- [1]Is Sycophancy a Form of In-Context Updating?arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 02:59 PM
We argue that this framing over-emphasizes the user and propose viewing sycophancy as a failure within the broader and generally useful process of in-context updating, by which models revise their answers based on information supplied during an interaction.
Factual sycophancy, a language model's deference to a user's false beliefs, is usually understood as a bid for the user's approval at the expense of truthfulness.
Extractive summary: sentences quoted from the sources.