DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge
We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure.
ProofPaper ↗
Key points
- Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer.
- Later cut-offs support evaluation of future models.
- We find that (1) single-shot retrieval significantly raises CAR in 38 of 40 configurations (model and thinking setting) versus no retrieval.
- Adding query rewriting and reranking to hybrid (dense & sparse) retrieval lowers CAR for all 12 models tested.
Sources (1)
- [1]DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric KnowledgearXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 03:07 PM
We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure.
Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Temporal Evidence-Parameter Co-Alignment for LLM Reasoning over Temporal Knowledge Graphs
- Oct 20, 2026Hallucinations of Large Language Models in Cross-Language Code Translation: Phenomena and Mitigation
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
- Oct 19, 2026Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
- Oct 19, 2026Honeyed Words, Hidden Daggers: Benchmarking and Training LLMs to See Through Mock Politeness
- Oct 19, 2026NADS: Navigator-Guided Data Selection for Mitigating Catastrophic Forgetting in Fine-Tuning