PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems

Key points

• Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable.1
• As a result, models often generate executable SQL even for queries with flawed premises.1
• To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries.1
• Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals.1

Sources (1)

• [1]PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 04:23 PM

  We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL.

  Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable.

Extractive summary: sentences quoted from the sources.