Progress-Guided Preference Generation for Aligning Generative Control Policies

Key points

• Generative policies have achieved strong performance in imitation learning tasks, but their success is often constrained by the limited availability of expert demonstrations.1
• Our method leverages ordered coverage alignment to measure sequential progress relative to a small set of expert demonstrations, enabling the automatic extraction of pseudo-preference labels from autonomously collected rollouts.1
• Furthermore, we introduce key enhancements to improve the reliability of these progress-guided preferences, enabling effective fine-tuning of pretrained generative policies without requiring manually designed reward functions.1
• Extensive experiments across a suite of challenging long-horizon manipulation tasks demonstrate that PGPG outperforms standard supervised fine-tuning baselines.1

Sources (1)

• [1]Progress-Guided Preference Generation for Aligning Generative Control Policies
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 02:38 AM

  In this work, we introduce Progress-Guided Preference Generation (PGPG), an unsupervised preference alignment framework that eliminates the need for human-in-the-loop supervision.

  Generative policies have achieved strong performance in imitation learning tasks, but their success is often constrained by the limited availability of expert demonstrations.

Extractive summary: sentences quoted from the sources.