From Association to Decision Value: Testing Paired Instruction Dependence for VLA Corpus Selection

Key points

• We study paired instruction dependence, the rate at which a demonstrator changes its first decision when only the goal changes.1
• Across 136 independently seeded quadruped-navigation corpora, predictions developed on 56 corpora are evaluated prospectively on 80 held-out corpora.1
• An exploratory response decomposition suggests a structural source of this predictive-ranking mismatch: under the tested predictor specification, separating both-correct and both-wrong responses yields larger held-out gains than their unsigned switch statistic.1
• These results show that counterfactual response structure can be reliable and predictive while providing limited incremental decision value in this setting when compressed into an unsigned statistic, motivating diagnostics that preserve outcome-relevant structure and are validated against the decisions they are intended to support.1

Sources (1)

• [1]From Association to Decision Value: Testing Paired Instruction Dependence for VLA Corpus Selection
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 12:27 PM

  Choosing among robot demonstration corpora is a ranking problem: an upstream diagnostic must preserve which candidates produce better downstream policies, not merely correlate with average performance.

  We study paired instruction dependence, the rate at which a demonstrator changes its first decision when only the goal changes.

Extractive summary: sentences quoted from the sources.