Stochastic Pruning of the DPO Computation Graph: 1.4x Faster Training and Lower Peak Memory

Key points

• Direct preference optimization (DPO) incurs training costs in output scoring, sequence execution, and backpropagation, so reducing supervision alone does not necessarily save computation.1
• Our analysis characterizes the objective changes induced by position and sequence pruning and uses inverse-probability weighting to obtain a conditionally unbiased single-branch gradient estimator for the sampled objective.1
• Controlled measurements show that position and sequence pruning primarily reduce peak memory, with most output-memory savings attributable to an implementation that retains full response supervision.1
• Branch pruning yields a $1.426\times$ step-time speedup over a concatenated DPO baseline and reduces training time by about 30% relative to stock DPO across two model families, with slightly lower mean accuracy and benchmark-dependent changes in open-ended scores.1

Sources (1)

• [1]Stochastic Pruning of the DPO Computation Graph: 1.4x Faster Training and Lower Peak Memory
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 10:25 AM

  We introduce a unified framework for stochastic pruning of the DPO computation graph that independently samples response positions for scoring, response prefixes for execution, and preference branches for backpropagation.

  Direct preference optimization (DPO) incurs training costs in output scoring, sequence execution, and backpropagation, so reducing supervision alone does not necessarily save computation.

Extractive summary: sentences quoted from the sources.