Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability

Key points

• This coupling can be degraded during training through, for example, CoT compression, optimization against a monitor, and out-of-context generalization from synthetic documents.1
• In this work, we propose an unsupervised post-training objective based on maximizing mutual information between CoT and the output for improving CoT monitorability.1
• Applied to undegraded policies, it further improves judged monitorability, improves prediction of unsafe behavior by CoT monitors and activation probes, and increases the prevalence of consequential reasoning.1
• A dedicated monitorability post-training stage can extend current practices that protect CoT from direct optimization pressure.1

Sources (1)

• [1]Monitorability Is Trainable: Post-Training for Chain-of-Thought Monitorability
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 01:35 PM

  Chain-of-thought (CoT) monitoring is one of the few practical tools available for overseeing capable AI systems, but it is fragile: a model's reasoning is only useful to a monitor if it remains coupled to what the model actually does.

  This coupling can be degraded during training through, for example, CoT compression, optimization against a monitor, and out-of-context generalization from synthetic documents.

Extractive summary: sentences quoted from the sources.