Reward Misspecification Degrades Chain-of-Thought Monitorability
Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety.
ProofPaper ↗
Key points
- As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought.
- One such practice may be unavoidable: reward misspecification.
- We show that reward misspecification alone already degrades monitorability.
- We create a testbed with pairs of true reward aligned with task intent and misspecified rewards that favor a different behavior, so that reward hacking can be induced by training and labeled verifiably.
Sources (1)
- [1]Reward Misspecification Degrades Chain-of-Thought MonitorabilityarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 02:45 PM
Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety.
As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking
- Oct 20, 2026Answering in the User's Variety: Calibrated Accommodation Pairs for Post-Training
- Oct 20, 2026From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks
- Oct 19, 2026Sparse RL Relaxation for LLM Reasoning
- Oct 19, 2026HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL
- Oct 19, 2026TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning