Asymmetric On-Policy Distillation Learns In-Context Exploration

Key points

• Sequential decision-making under partial observability is difficult because agents must explore to infer hidden, task-relevant information before acting effectively.1
• Learning such information-gathering behaviors tabula rasa via online reinforcement learning is sample-inefficient, whereas offline imitation averages expert actions across hidden states and fails to learn exploration.1
• Although a privileged expert never needs to explore, we demonstrate that a history-conditioned student can learn to do so when trained with on-policy contexts: the student first collects on-policy context under partial observability, and the privileged expert provides action labels conditioned on that context.1
• This enables efficient test-time exploration (similar to Bayesian posterior sampling) without incurring the impractical training-data costs of reinforcement learning.1

Sources (1)

• [1]Asymmetric On-Policy Distillation Learns In-Context Exploration
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:50 PM

  In this work, we propose ASTEROID, a simple and effective student--teacher supervised learning framework that can learn test-time exploration.

  Sequential decision-making under partial observability is difficult because agents must explore to infer hidden, task-relevant information before acting effectively.

Extractive summary: sentences quoted from the sources.