Steered Not Stirred: Effective Reward Tilting using Surrogate Reward Mixing

Key points

• Continuous-time generative models, from diffusion models, flow matching, and now flow maps, owe their practical value to high-fidelity generation and steerability.1
• In this paper, we present a method for composing such surrogates to steer a pretrained model toward the distribution tilted by an unavailable $R^$.1
• Our theory characterizes surrogate information limits and relates population recovery error to ideal-target KL at small tilt strength.1
• Experiments demonstrate improved oracle-tilt fidelity in language generation and ImageNet alignment, and improved joint pose accuracy and validity in protein-ligand cofolding, without inference-time oracle queries.1

Sources (1)

• [1]Steered Not Stirred: Effective Reward Tilting using Surrogate Reward Mixing
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 07:42 PM

  Given a small feedback set of oracle rankings or numerical rewards, we introduce Mixing And Reward Tilting for INference-time Improvement (MARTINI), which learns a surrogate composition for plug-and-play use with standard inference-time steering methods.

  Continuous-time generative models, from diffusion models, flow matching, and now flow maps, owe their practical value to high-fidelity generation and steerability.

Extractive summary: sentences quoted from the sources.