Statistical Learning Theory for Direct Preference Optimization

Key points

• We study how failure of this modeling assumption limits DPO, distinguishing structural misspecification from limited model capacity and finite-sample estimation.1
• Our main contribution is a characterization of structural approximation error through the triangle-curl of the preference log-odds, which vanishes exactly when preferences admit a Bradley--Terry representation.1
• We further establish finite-sample bounds for clipped empirical risk minimizers that separate network approximation, structural misspecification, and estimation error.1
• We place these results within a surrogate-risk framework by identifying contextual preference accuracy (CPA) as a natural measure of agreement between preference judgments and changes in pairwise probability ratios relative to the reference policy.1

Sources (1)

• [1]Statistical Learning Theory for Direct Preference Optimization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 06:12 AM

  Direct Preference Optimization (DPO) learns from pairwise preferences through a logistic objective whose score-difference parameterization follows the Bradley--Terry model.

  We study how failure of this modeling assumption limits DPO, distinguishing structural misspecification from limited model capacity and finite-sample estimation.

Extractive summary: sentences quoted from the sources.