Claude’s new auto eval tool
Anthropic released new eval tooling for Claude Code.
Proof1 independent outlet
Key points
- Their claude-api plugin now includes a new buildeval and hill-climb command that helps you build evals, check the graders, and improve your application against them.
- I usually don’t review eval tools because software changes so often that reviews have a short shelf life.
- But a first-party tool from Anthropic is likely to influence how people approach evals, so I wanted to try it and share what I found.
- In the screenshot below, Claude asks us to skim inputs.md and “tell it” which labels are wrong.
Sources (1)
- [1]Claude’s new auto eval toolHamel Husain · Sep 30, 07:00 AM
Anthropic released new eval tooling for Claude Code.
Their claude-api plugin now includes a new build_eval and hill-climb command that helps you build evals, check the graders, and improve your application against them.
Extractive summary: sentences quoted from the sources.
Hype check
Independently evaluated1 independent look since
- Evaluation · Oct 2, 2026openqodex/openqodex: Open source AI code review for Claude Code and Codex, before you push. Scanners (SAST, secrets, dependencies, lint) on the lines you changed
- Follow-up · Sep 30, 2026Last Week in AI #345 - 5 new models, 9 misalignment incidents, some Dots
- Follow-up · Sep 30, 2026nanaism/yomiyasu: AI生成の日本語を自然な日本語へ推敲するAgent Skill / Agent Skill for Refining AI-Generated Japanese into Natural Japanese
- Follow-up · Sep 30, 2026justxor/claude-code-pro-course: Claude Code PRO — полный курс и гайд на русском: CLAUDE.md, settings, skills, subagents, hooks, MCP, плагины, CI. 8 лабораторных, библиотека
- Follow-up · Sep 30, 2026Edwardxlai/easyread: 把英文论文读成舒服的中文:本地 PDF 论文翻译、原文对照、边读边问 AI、文献管理。Read English papers in comfortable Chinese.
- Follow-up · Sep 30, 2026Cloudflare Containers, rebuilt to scale agent sandboxes
Before this
- Sep 29, 2026langchain-ai/langchain langchain-anthropic==1.7.5
- Sep 29, 2026Sonnet 5.5 is worth a try
- Sep 29, 2026[AINews] AMD buys World Labs for $8.2B, as Atlas solves sparse reconstruction problem for robotics, design and more
- Sep 29, 2026Claude Code’s Next Era — Thariq Shihipar, Anthropic
- Sep 29, 2026What do you want from AI?
- Aug 21, 2026ollama/ollama v0.33.0
