FDE-Gym: The Last Mile of AI Delivery
We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects.
ProofPaper ↗
Key points
- Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets.
- Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints.
- In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions.
- Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance.
Sources (2)
- [1]FDE-Gym: The Last Mile of AI DeliveryarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 05:57 PM
We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects.
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets.
- [2]FDE-Gym: The Last Mile of AI DeliveryHugging Face Daily Papers · Oct 15, 12:00 AM · same content
Extractive summary: sentences quoted from the sources.
Before this
- Oct 14, 2026Two-Stage Score Estimation for Group-Specific Diffusion Models
- Oct 14, 2026DefoliationCAM: An Edge-AI System for Real-Time Insect Defoliation Severity Assessment in Grapevines
- Oct 14, 2026Long-Context Drafting Does Not Require Long-Context Drafters
- Oct 14, 2026When Reasoning Traces Conflict with Evidence: How Answer Configuration Shapes Evidence Following
- Oct 14, 2026mistralai/Timestral-209M
- Oct 14, 2026LEV: Thinking Fast and Deep with Looped Decision Models