ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes

Key points

• Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning).1
• However, scaling IFs to LLMs is hindered by the intractability of billion-scale Hessian matrices and the prohibitive memory overhead of backpropagation (BP).1
• ZOIF decomposes influence estimation into three BP-free stages: (1) approximating the average proxy gradient via zeroth-order estimation; (2) reformulating and solving the inverse Hessian-vector product as an auxiliary zeroth-order optimization to bypass explicit inversion; and (3) deriving per-sample influence scores via a forward finite difference along the previously computed parameter update direction.1
• Experiments show that ZOIF selects data subsets with downstream utility competitive with, and in some cases exceeding, the evaluated baselines, yet at substantially lower peak memory cost and with comparable overall computation time.1

Sources (1)

• [1]ZOIF: Zeroth-Order Influence Function Estimation for LLM Data Selection Using Only Forward Passes
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 11, 04:25 AM

  To address this, we propose ZOIF, a zeroth-order framework for influence estimation that operates entirely via forward passes, enabling memory-efficient data selection under resource constraints.

  Influence functions enable principled data selection for large language models (e.g., identifying influential training samples for model editing, debugging, or efficient fine-tuning).

Extractive summary: sentences quoted from the sources.