Probabilistic Multi-Source Data Selection via Learnable Categorical Priors: Variance Reduction and Guaranteed Derandomization

Key points

• Bilevel data selection is a fundamental task with numerous applications in machine learning and statistics.1
• Our key idea is to introduce multiple learnable categorical priors to parameterize a probability distribution supported only on feasible data subsets, thus naturally yielding a lossless rounding scheme.1
• We further show that our proposed relaxation admits an effective two-point gradient estimator, which can simultaneously avoid the inverse-Hessian computation required by existing 0-1 relaxation and the unbounded score term appearing in prior probabilistic relaxation.1
• Built upon this reducible two-point estimator, we then design a novel stochastic gradient algorithm, termed ReSGD, for multi-source bilevel data selection problem, and establish a rigorous approximation guarantee for the subset returned by ReSGD under weak submodularity.1

Sources (1)

• [1]Probabilistic Multi-Source Data Selection via Learnable Categorical Priors: Variance Reduction and Guaranteed Derandomization
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 02:14 PM

  To address these issues, this paper proposes a new probabilistic relaxation for the multi-source bilevel data selection problem.

  Bilevel data selection is a fundamental task with numerous applications in machine learning and statistics.

Extractive summary: sentences quoted from the sources.