Robust Direct Preference Optimization under Preference Label-Flip Attacks

Key points

• However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy.1
• To address this issue, we design robust DPO methods that can defend against such attacks.1
• Building on the property that lable flips induce parameter-independent gradient shifts in log-linear DPO, we propose Trim-$K$ Robust-DPO, which removes the $K$ highest-impact pairs, and Soft-Trim-$K$ Robust-DPO, which smoothly downweights them.1
• We establish deterministic gradient perturbation bounds uniformly over all attack sets containing at most $K$ flips, explicitly separating defense-induced bias from surviving attack effects.1

Sources (1)

• [1]Robust Direct Preference Optimization under Preference Label-Flip Attacks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 01:38 AM

  Direct Preference Optimization (DPO) is widely used for offline Reinforcement Learning from Human Feedback.

  However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy.

Extractive summary: sentences quoted from the sources.