What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports
We conduct CTXGAP, a measurement study quantifying the gap in task-state recoverability between agents' accumulated user-facing reports and their full recorded execution contexts.
ProofPaper ↗
Key points
- Human supervisors rely on coding agents' reports to assess progress, verify repairs, and decide when to intervene.
- Across 37 coding tasks and 4,799 valid probes, agents answer 93.6% of task-state questions on demand; full-context readers recover 92.7%, while report-only readers recover 52.0%, a 40.7-point gap.
- Human annotation identifies states mentioned without enough supporting detail as the largest category of unrecovered information, and model-assessed supervision-relevant states retain substantial gaps.
- Positive gaps persist across harnesses and independent DeepSeek execution and measurement.
Sources (1)
- [1]What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and ReportsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 11:20 AM
We conduct CTXGAP, a measurement study quantifying the gap in task-state recoverability between agents' accumulated user-facing reports and their full recorded execution contexts.
Human supervisors rely on coding agents' reports to assess progress, verify repairs, and decide when to intervene.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning
- Oct 13, 2026Thank-you post for DeepSeek-V4-Flash-0731 with DSpark
- Oct 13, 2026MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization
- Oct 13, 2026Auditing Future-Memory Use in Point-in-Time Language Agents
- Oct 13, 2026AssetForge: Asset-Augmented Task Synthesis for Business Automation Agents