Generative Policy Improvement from Observational Data

Key points

• We study how to improve decision policies from observational data to generate better actions when actions are complex, structured objects, such as text messages or images.1
• This task is subject to two major challenges: (i) action-level overlap is often too weak for reliable off-policy evaluation and improvement, and (ii) generating new structured actions often requires expensive training or fine-tuning (e.g., of an LLM) and can be ill-defined due to information loss in action representations.1
• We show theoretically that improving the policy over action codes guarantees improvement of the induced policy in the original action space.1
• We demonstrate PolicyLift across experiments in synthetic and real-world text generation tasks with consistent policy improvements over the behavior policy.1

Sources (1)

• [1]Generative Policy Improvement from Observational Data
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 01:20 PM

  To address both, we propose PolicyLift, a three-stage framework for generative policy improvement with structured actions from observational data.

  We study how to improve decision policies from observational data to generate better actions when actions are complex, structured objects, such as text messages or images.

Extractive summary: sentences quoted from the sources.