Understanding and Reducing Misalignment in Persuasion Optimization
In this work, we study how reinforcement learning (RL) training for persuasion affects models' use of misaligned strategies such as evidence fabrication and belief coercion, and what techniques can reduce such misalignment.
ProofPaper ↗
Key points
- Frontier models are now trained for complex multi-agent tasks, from running a company to engaging in negotiations.
- But what does this optimization pressure mean for model alignment?
- We find that optimizing sender models solely for receiver belief shifts causes the average number of distinct misaligned strategies used per game to increase by more than 25%, and fabrications per game increase to more than 4 times relative to pre-RL models.
- We find that navigating the persuasiveness-misalignment trade-off in this setting is challenging, with the best trade-offs coming from combining guidance on aligned persuasion strategies with reward penalties on misalignment.
Sources (1)
- [1]Understanding and Reducing Misalignment in Persuasion OptimizationarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 02:19 AM
In this work, we study how reinforcement learning (RL) training for persuasion affects models' use of misaligned strategies such as evidence fabrication and belief coercion, and what techniques can reduce such misalignment.
Frontier models are now trained for complex multi-agent tasks, from running a company to engaging in negotiations.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning
- Oct 12, 2026StartLux-Decision: AutoThink Decision Models with Reinforcement Learning for Calibrated Deliberative Decisions
- Oct 12, 2026LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards
- Oct 11, 2026AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards
- Oct 11, 2026RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
- Oct 11, 2026ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs
