One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents

Key points

• Recent advances in multi-turn reinforcement learning (RL) have substantially improved the performance of LLM agents on complex, long-horizon interactive tasks.1
• However, despite progress in fine-grained credit assignment, policy optimization still applies these learning signals through response-specific probability ratios.1
• Different textual responses can induce the same executable effect, yet credit differences arising from subsequent outcomes remain tied to their individual textual realizations.1
• Our analysis reveals that credit-text pairing directly shapes policy-update directions, even when the policy context and executable effect are held fixed.1

Sources (1)

• [1]One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 08:51 AM

  Motivated by this finding, we propose Effect-Marginal Policy Optimization (EMPO), which aligns policy optimization with execution-level equivalence through an effect-level surrogate that couples updates across alternative textual realizations.

  Recent advances in multi-turn reinforcement learning (RL) have substantially improved the performance of LLM agents on complex, long-horizon interactive tasks.

Extractive summary: sentences quoted from the sources.