Differentially Private Sparse Reward Estimation with Preference Feedback

Key points

• Reinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values.1
• Existing private alignment methods typically assume dense reward parameterizations, resulting in estimation error bounds that scale polynomially with the ambient {model size $d$}.1
• In this work, we study Private Sparse RLHF problem to address this limitation, assuming that human preferences (reward model) are governed by a subset of $s^ \ll d$ relevant features.1
• For the lower bounds, we show that the Sample-DP rate is nearly optimal, and that the guarantee for our Label-DP method cannot be further improved within the randomized-response framework.1

Sources (1)

• [1]Differentially Private Sparse Reward Estimation with Preference Feedback
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 05:45 PM

  We provide the first comprehensive theoretical framework for differentially private sparse reward estimation from pairwise preference feedback under the Bradley-Terry-Luce model, and introduce efficient algorithms for both Sample-DP and Label-DP settings.

  Reinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values.

Extractive summary: sentences quoted from the sources.