The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Key points

• However, when this behaviour develops during training and what a released model inherits from earlier stages remain unclear.1
• We follow 741 checkpoints of 30 open model lineages, from early pretraining to released models.1
• Beyond responses without pressure, pretraining behaviour predicts little of which questions a released model yields on.1
• That pattern forms during post-training: on routes with supervised fine-tuning, the fine-tuning checkpoint predicts part of it and the preference-optimization checkpoint most of it.1

Sources (1)

• [1]The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 04:13 PM

  Language models can change a committed answer to accommodate a user's opinion or pushback without new evidence.

  However, when this behaviour develops during training and what a released model inherits from earlier stages remain unclear.

Extractive summary: sentences quoted from the sources.