How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
Nathanael Weill leads a team building AI agents at Air Canada, where tasks are so specific there's no public benchmark and domain experts have little time to grade results.
Proof1 independent outlet
Key points
- His fix is to write the answer first: start from a known answer, generate the input from it, and check whether the agent finds its way back.
- He shows how that took a past RAG project from one expert-graded test a week to dozens of automated tests a day, then applies it to a supplier-contract review agent with five ways to build a test set: past decisions, planted rule violations, removed clauses, changes that shouldn't matter, and contract templates with filled-in blanks.
- 10:08 How the contract agent works
- 18:07 Use expert feedback to build better tests
Sources (1)
- [1]How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air CanadaAI Engineer (YouTube) · Oct 22, 08:00 AM
Nathanael Weill leads a team building AI agents at Air Canada, where tasks are so specific there's no public benchmark and domain experts have little time to grade results.
His fix is to write the answer first: start from a known answer, generate the input from it, and check whether the agent finds its way back.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Securing AI agent identity with X.509 PKI on AWS
- Oct 20, 2026Temporal Evidence-Parameter Co-Alignment for LLM Reasoning over Temporal Knowledge Graphs
- Oct 20, 2026Voxtral 3
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 20, 2026Toward Agentic AI for the 6G Physical Layer: From Algorithm Discovery to Verified Baseband Silicon
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
