Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards

Key points

• Runtime policy verifiers can prevent agents from executing disallowed actions, but their decisions are typically used only for enforcement and discarded afterward.1
• We ask whether these decisions can instead become verifiable policy rewards for training the agents they guard.1
• Compliant trajectories are reinforced, and violations become negatives for preference optimization; beyond the gate and the benchmark's task-success signal, PVST needs no stronger teacher model, no human preference labels and no attack-specific supervision, although rollouts are collected in the benchmark's attacked environment.1
• The learned behavior holds under an attack template never used in training (37.7% to 11.0%) and on a suite excluded from training (34.8% to 16.7%), and the retained verifier's intervention rate falls from 39.2% to 8.8% (seed 0).1

Sources (1)

• [1]Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 01:16 AM

  We introduce Policy-Verified Self-Training (PVST), a post-training paradigm in which an agent generates its own interaction trajectories and an executable policy verifier, here a deterministic gate compiled from a written policy, evaluates whether each action complies.

  Runtime policy verifiers can prevent agents from executing disallowed actions, but their decisions are typically used only for enforcement and discarded afterward.

Extractive summary: sentences quoted from the sources.