When is a better LLM worth paying for? Stochastic Optimization with Switching Mixed-Fidelity Oracles

Key points

• Large language models (LLMs) are increasingly used upstream of decision-making to clean data, detect label errors, and reconstruct missing values.1
• We study this question through stochastic optimization with mixed-fidelity first-order oracles, which can have systematic bias and state-dependent variance.1
• We develop two cost-aware algorithmic frameworks that allow adaptive switching between stochastic oracles.1
• With unknown bias, we develop a budgeted certification scheme that uses occasional trusted-gradient evaluations to certify the optimization progress and eliminate unsuccessful oracle sources, and show an explicit tradeoff between certification cost and delayed detection of biased sources.1

Sources (1)

• [1]When is a better LLM worth paying for? Stochastic Optimization with Switching Mixed-Fidelity Oracles
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 02:50 AM

  Yet LLM-assisted data curation creates a new operational tradeoff: higher-fidelity models and richer prompts can be substantially more expensive, while the value of this additional fidelity depends on the downstream optimization accuracy.

  Large language models (LLMs) are increasingly used upstream of decision-making to clean data, detect label errors, and reconstruct missing values.

Extractive summary: sentences quoted from the sources.