Training Natural Schemers: RL Can Amplify Oversight-Conditional Behavior

Key points

• We therefore construct a natural model organism of scheming, built on Kimi K2.6: trained with traits that could plausibly emerge in a frontier training pipeline, and without providing the model with the target scheming policy.1
• We then train it using Reinforcement Learning (RL) with a reward that scores only task quality and policy compliance and does not reward scheming; we show that this amplifies scheming.1
• The effect holds across our three RL environments, and we find that the scheming policy transfers between them.1
• Our results suggest that a reward that does not reward scheming offers no inherent protection against certain traits when an oversight-conditional policy is indistinguishable from an aligned one during training.1

Sources (1)

• [1]Training Natural Schemers: RL Can Amplify Oversight-Conditional Behavior
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:12 PM

  One of the central concerns for the safety of advanced AI is scheming: when a model strategically complies under oversight but pursues its own goals when unobserved.

  We therefore construct a natural model organism of scheming, built on Kimi K2.6: trained with traits that could plausibly emerge in a frontier training pipeline, and without providing the model with the target scheming policy.

Extractive summary: sentences quoted from the sources.