Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards

Key points

• We propose shifting this burden to the robot.1
• Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering.1
• We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features.1
• In simulation, the informed prior beats the standard active preference learning baseline by up to $30.8%$ in the low-query regime, where reducing human effort matters most.1

Sources (1)

• [1]Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:41 PM

  Reward learning typically assumes the human's input covers everything that matters for the task.

  We propose shifting this burden to the robot.

Extractive summary: sentences quoted from the sources.