EvalRx: Agentic Diagnosis and Verified Repair of Foundation Models

Key points

• Model evaluation reveals where models fail, but rarely why or whether a proposed fix genuinely works.1
• EvalRx discovers systematic failure modes, explores candidate mechanisms, formulates falsifiable hypotheses, and verifies them by executing frozen, pre-specified tests before using supported diagnoses to guide tiered repair.1
• Across language, vision-language, and audio-language settings spanning Gemini, Qwen, Gemma, and Nemotron, EvalRx improves performance in all 12 evaluated model family-modality combinations, with gains ranging from +1.4 to +10.9 percentage points (+5.8 points on average).1
• These results show that model evaluation can serve not only to measure failures, but to support an auditable process for diagnosing and repairing them.1

Sources (1)

• [1]EvalRx: Agentic Diagnosis and Verified Repair of Foundation Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 04:28 AM

  We introduce EvalRx, an open-source agentic framework for statistically grounded model diagnosis and repair.

  Model evaluation reveals where models fail, but rarely why or whether a proposed fix genuinely works.

Extractive summary: sentences quoted from the sources.