Tracing Recurrent and Emergent Hallucination Patterns in Decomposed Text-to-SQL Generation

Key points

• Large language models can achieve strong Text-to-SQL benchmark performance while still producing schema and query-logic errors.1
• We distinguish recurrent hallucinations, in which the same rule-based category is observed in an earlier step and again in the final query, from emergent hallucinations, in which the category is observed at the final step without an earlier detection in the same path.1
• Across five LLMs evaluated on an expanded BIRD-mini benchmark, several categories show appreciable recurrence probabilities, while clause abuse shows comparatively higher conditional emergence.1
• Under the conversational-history condition, recurrence probabilities are generally higher and emergence probabilities are generally lower.1

Sources (1)

• [1]Tracing Recurrent and Emergent Hallucination Patterns in Decomposed Text-to-SQL Generation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:39 AM

  We present a drill-down evaluation framework that decomposes Text-to-SQL queries into progressively more complex executable sub-queries and corresponding natural-language questions, allowing rule-based hallucination categories to be examined across a generation path.

  Large language models can achieve strong Text-to-SQL benchmark performance while still producing schema and query-logic errors.

Extractive summary: sentences quoted from the sources.