When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks

Key points

• Web agents increasingly browse, interpret, and act on web content on behalf of users.1
• In many realistic workflows, however, the visual evidence they consume is not neutral: charts and dashboards can be misleading, shaping judgments without issuing any explicit malicious instruction.1
• Unlike chart question answering, Web Agents treat charts as intermediate evidence whose misinterpretation can propagate into concrete downstream actions --- a reliability risk that remains underexplored.1
• These results reveal that visualization robustness is a critical yet understudied dimension of Web Agent safety and reliability.1

Sources (1)

• [1]When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 12:03 AM

  We introduce MisVisAgentBench, the first paired executable benchmark for evaluating how misleading visualizations affect Web Agent execution.

  Web agents increasingly browse, interpret, and act on web content on behalf of users.

Extractive summary: sentences quoted from the sources.