When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring

Key points

• We study a fixed vision-language model (VLM) that judges success while a policy improves.1
• In a controlled 20-seed tray-placement experiment, training selects only simulator-verified successes.1
• We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge.1
• Judge-filtered experiments across eight settings further show heterogeneous effects of state-dependent acceptance.1

Sources (1)

• [1]When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 06:08 PM

  Can a better robot policy become harder to monitor?

  We study a fixed vision-language model (VLM) that judges success while a policy improves.

Extractive summary: sentences quoted from the sources.