From Outcomes to Steps: Reliable Success-Aware Critic Learning for Long-Horizon Language Agents

Key points

• The main difficulty is that a critic with aggregate return-prediction accuracy may still fail to resolve the local value differences needed to assign credit to actions, especially under sparse feedback and function approximation.1
• We introduce SCALE, an actor-critic framework that augments conventional return estimation with explicit prediction of eventual task success.1
• At each interaction prefix, the critic predicts a return value and a Success Value, defined as the probability that the task will eventually be completed from that prefix.1
• Separate critic-learning analyses show that SCALE also improves critic estimation and training stability.1

Sources (1)

• [1]From Outcomes to Steps: Reliable Success-Aware Critic Learning for Long-Horizon Language Agents
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:40 AM

  Reinforcement learning for long-horizon language agents requires identifying whether intermediate actions make eventual task success more or less likely when supervision is available only at trajectory level.

  The main difficulty is that a critic with aggregate return-prediction accuracy may still fail to resolve the local value differences needed to assign credit to actions, especially under sparse feedback and function approximation.

Extractive summary: sentences quoted from the sources.