NLData2Opt: Benchmarking Parameter Derivation and Prediction for Modeling Optimization Problems from Natural Language and Data

Key points

• Formulating optimization problems from raw inputs of problem descriptions and data is time-consuming and costly.1
• While Large Language Models (LLMs) offer a promising way to automate this process, existing benchmarks overestimate their capabilities by relying on idealized inputs where all data is relevant and complete.1
• NLData2Opt decomposes problems into problem descriptions and data, and requires models to (1) extract relevant parameters from data containing misleading attributes, (2) compute missing parameters derived by transforming existing data attributes, and (3) predict unavailable future values from historical observations.1
• Evaluation results show substantial performance decline relative to existing benchmarks, indicating that benchmark progress does not directly translate to formulation accuracy in practical optimization workflows.1

Sources (1)

• [1]NLData2Opt: Benchmarking Parameter Derivation and Prediction for Modeling Optimization Problems from Natural Language and Data
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 03:43 AM

  To address this, we introduce NLData2Opt, a benchmark to evaluate LLMs on the multi-stage reasoning inherent in practical optimization workflows.

  Formulating optimization problems from raw inputs of problem descriptions and data is time-consuming and costly.

Extractive summary: sentences quoted from the sources.