SHAM-Bench: A Span-level Hallucination Benchmark Across Multiple Tasks

Key points

• Span-level hallucination detection pinpoints unsupported content in LLM outputs, enabling actionable feedback and targeted mitigation.1
• Existing evaluations for span-level hallucination detection methods are fragmented across tasks, annotation formats, and metrics, and most span-level datasets cover only a single NLG domain, making it difficult to assess whether a detection method actually generalizes across tasks and domains.1
• Using SHAM-Bench, we evaluate existing hallucination detection methods and find that fine-tuned detectors often perform well in-domain but degrade under distribution shift, whereas prompt-based methods with reasoning are more robust across tasks.1
• We release the dataset, evaluation code, and a public leaderboard to support future research on broadly generalizable hallucination detection.1

Sources (1)

• [1]SHAM-Bench: A Span-level Hallucination Benchmark Across Multiple Tasks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 10:48 PM

  In this paper, we introduce SHAM-Bench (Span-level Hallucination Benchmark Across Multiple Tasks), a unified benchmark for span-level hallucination detection spanning six NLG tasks, aggregated from existing human-annotated datasets totaling 6,461 examples under a common schema and evaluation protocol.

  Span-level hallucination detection pinpoints unsupported content in LLM outputs, enabling actionable feedback and targeted mitigation.

Extractive summary: sentences quoted from the sources.