TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization

Key points

• Direct Preference Optimization (DPO) minimizes empirical risk over observed preference comparisons, implicitly assuming that the training preference distribution matches the deployment preference distribution.1
• Such multi-level distributional mismatch poses a critical challenge for safe LLM deployment, yet existing robust DPO methods typically either model uncertainty over the joint preference distribution or focus on individual sources of shift, leaving the hierarchical structure of OOD preference shifts underexplored.1
• By instantiating it with Wasserstein distance, we derive a tractable and interpretable objective that combines a closed-form preference-level margin penalty, prompt-level adversarial embedding perturbation, and worst-case group reweighting, each governed by an independent radius.1
• We establish finite-sample convergence guarantees and empirically validate the effectiveness of TPO over existing robust DPO methods.1

Sources (1)

• [1]TPO: Tri-level Distributionally Robust Learning for OOD Direct Preference Optimization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 01:57 PM

  We propose TPO, a tri-level distributionally robust framework for OOD preference alignment, which jointly models all three levels of distributional uncertainty through a hierarchically structured rectangular ambiguity set.

  Direct Preference Optimization (DPO) minimizes empirical risk over observed preference comparisons, implicitly assuming that the training preference distribution matches the deployment preference distribution.

Extractive summary: sentences quoted from the sources.