ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence

Key points

• Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system rather than merely answer a prompt.1
• The key idea is to separate the readable scientific surface from the formal causal world: LLMs generate natural-language scenarios and task text, while hidden probabilistic graphical models define the causal graph, data-generating process, intervention semantics, validation checks, and oracle answers.1
• We evaluate seven contemporary LLMs and two earlier-generation models across three active agent designs.1
• Evidence-ledger analysis localizes failures within the investigation: agents test the wrong intervention, lose relevant evidence, or continue after sufficient evidence has appeared.1

Sources (1)

• [1]ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 11:01 PM

  We introduce ACED-BENCH (Active Causal Evidence and Decision Benchmark), a neuro-symbolic benchmark built around this requirement.

  Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system rather than merely answer a prompt.

Extractive summary: sentences quoted from the sources.