MisleadQA: Diagnosing LLM Failures under Misleading Context
We introduce MisleadQA, a diagnostic dataset of multiple-choice questions with an abstention option.
ProofPaper ↗
Key points
- Large language models (LLMs) are widely used to answer questions from retrieved or user-provided context.
- However, provided context can be factually true yet out-of-scope, describing a different entity, time, or location from the one the question targets, and still lead models to a wrong answer.
- It remains unclear whether such out-of-scope context misleads models as much as false context, and whether adding gold context prevents models from being misled by either.
- Each question pairs gold context (G) with true but out-of-scope context (S) and false but in-scope context (F) supporting the same wrong answer.
Sources (1)
- [1]MisleadQA: Diagnosing LLM Failures under Misleading ContextarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 07:58 PM
We introduce **MisleadQA**, a diagnostic dataset of multiple-choice questions with an abstention option.
Large language models (LLMs) are widely used to answer questions from retrieved or user-provided context.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 17, 2026Refuse the Harm, Complete the Task via Graph-of-Rules Planning for LLM-Enabled Robots
- Oct 16, 2026[Research] 2,788 tokens/s on a single RTX 5090 with a diffusion LLM — Introducing ORL (fully open source)
- Oct 16, 2026Perceive, Solve, Reason: Progressive Capability Formation in Time Series-LLMs
- Oct 16, 2026Language Models Learn to Improve Language Models
- Oct 16, 2026On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs
- Oct 15, 2026Agentic-Priors: Using Large Language Models to Address the Cold Start Problem in Preference Elicitation