The AI Agent Evaluation Handbook
Everything I know about proving an AI agent works — organized as one map. Each chapter distills a topic and links the deep-dive: scenario design, tool-use benchmarking, judge calibration, CI gates, and how to staff it. Written from real benchmark work: 78 scenarios across 43 tools, evaluation pipelines processing 100K+ judgments a day, and tasks shipped into a published frontier-lab benchmark.
Last updated: July 28, 2026
What AI evaluation actually is
Evaluation is not "trying prompts and eyeballing outputs." It is a repeatable system: representative scenarios, explicit pass/fail criteria, and scoring you can run on every release. Demo prompts systematically overstate quality — cherry-picked inputs, narrow distributions, no adversarial pressure — which is why teams that test by demo get ambushed in production.
Deep dive: Why Demo Prompts Lie About Model QualityFramework or custom harness?
Open-source frameworks (DeepEval, Ragas, LangSmith evals) are excellent for standard metrics on single-turn outputs — use them. They are not, by themselves, an evaluation system for your product: scenario design, tool-call validation, and domain-calibrated judges still have to be built. The decision comes down to what you are testing and what is at stake.
Deep dive: DeepEval vs. a Custom Eval HarnessDesigning eval scenarios that predict production
A useful eval set is stratified like real traffic: common cases, edge cases, adversarial inputs, and failure-recovery paths — each with unambiguous scoring criteria decided before you run anything. A practical framework: tier scenarios from single-tool happy paths up to ambiguous multi-tool tasks, and set per-tier launch thresholds.
Deep dive: How to Evaluate an AI Agent Before LaunchBenchmarking tool-use and MCP agents
Agents fail differently from chatbots: wrong tool selection, malformed parameters, broken multi-step chains, poor recovery from tool errors. Benchmarking them means deterministic validation of every call — right tool, right arguments, right order — plus scenario tiers for ambiguity. Tool-selection accuracy degrades non-linearly as tool count grows; parameter formatting, not tool choice, is the most common failure mode.
Deep dive: How to Benchmark Tool-Use in AI AgentsLLM-as-a-Judge, without lying to yourself
Judges scale scoring for open-ended outputs, and they carry biases: position, verbosity, self-preference, sycophancy. They work when criteria are structured and calibrated against human labels — and mislead when you point them at open-ended "quality" without anchors. Rubric vs. pairwise, calibration loops, and spot-check sampling are the difference.
Deep dive: LLM-as-a-Judge: When It Works and When It LiesStress-testing frontier models
Every model has a breaking surface: constraint stacking, negation, temporal reasoning, format compliance under pressure, long-output self-consistency. Systematic adversarial testing finds it before your users do — and every found failure becomes a regression test with pinned seeds, temperatures, and versioned prompts.
Deep dive: What Breaks Frontier ModelsWiring evals into CI
An eval you run once is a report; an eval that runs on every model swap, prompt change, and dependency bump is a quality gate. That means baselines, tolerance bands, nightly full runs with PR-time subsets, and dashboards someone actually looks at.
Deep dive: Agent Benchmarking Services (CI-ready harnesses)Who does the work: hire, borrow, or build?
Full-time evals engineers are scarce and expensive; generalist freelancers rarely have the specialty; fixed-scope specialists get you a working system fast that your team then owns. The honest comparison depends on your stage and how often evaluation needs change.
Deep dive: Hiring an AI Evals Engineer: the options compared