The technical questions engineers actually ask about evaluating AI agents — answered with numbers measured on real harnesses, not general advice. Every figure here traces to a published benchmark or a case study.
How many tools can an AI agent handle before tool selection degrades?
Tool-selection accuracy holds up to roughly 15 tools, then degrades non-linearly. In a 43-tool MCP benchmark, average selection accuracy measured 82% on single-tool tasks but fell to 54% on the hardest tier, and even the strongest models picked the wrong tool 15–20% of the time at 43 tools.
The degradation is not gradual. Below ~15 tools, models disambiguate reliably. Past that, overlapping tool descriptions (a CRM contact search next to an admin user search) start producing confident wrong selections. If your production agent exposes 30–60 tools — the normal range for enterprise deployments — you are operating in the region where accuracy falls apart, and single-tool demo testing will not reveal it.
What is the most common failure mode in tool-using AI agents?
Malformed parameters, not wrong tool choice. Agents frequently select the correct tool and then fail on the call: missing required fields, wrong data types (a string where a number is expected), invalid enum values, and hallucinated parameter names that do not exist in the schema.
Most teams test whether the agent picks the right tool and stop there. That misses the larger error class. Deterministic parameter validation — asserting the exact arguments against the tool schema on every call — catches these before users do, and it is cheap because it needs no model in the loop.
Coverage matters more than count. A production-grade reference harness used 78 scenarios stratified across five tiers: 15 single-tool, 18 sequential, 15 parallel, 18 conditional, and 12 adversarial. The tiering is the point — a hundred single-tool scenarios tell you less than fifty spread across difficulty levels.
Stratify by what can break rather than by feature. Single-call scenarios isolate selection and formatting. Multi-step scenarios test whether the agent carries real IDs forward instead of inventing plausible ones. Error-recovery scenarios require deliberately injected failures. Ambiguous-input scenarios check whether the agent asks a clarifying question instead of guessing.
How reliable is LLM-as-a-judge compared to human evaluation?
A well-calibrated judge reaches about 0.89 Spearman correlation with expert human labels at roughly 1/100th the cost of full human evaluation. That figure comes from a pipeline running 100K+ judgments a day — but it depends entirely on calibration against human labels, not on the judge model being strong.
Judges fail in predictable ways: position bias, verbosity bias, self-preference, and sycophancy. They work when criteria are structured with explicit anchors and validated against a human-labelled sample. They mislead when pointed at open-ended "quality" with no rubric. Re-calibrate after every model version bump — judge agreement drifts when the underlying model changes.
Why do AI features pass testing and then fail in production?
Demo prompts systematically overstate quality. They are cherry-picked, narrowly distributed, and never adversarial, so they measure the happy path and miss the failure surface. A representative eval set has to mirror real traffic: common cases, edge cases, adversarial inputs, and failure-recovery paths, each with pass/fail criteria fixed before the run.
The practical test: if your eval set was assembled by writing prompts you thought of, it is a demo set. If it was assembled from real user traffic, logged failures, and deliberately hostile inputs, it is an eval set. The second one predicts production; the first one predicts your own imagination.
Test the contract, not just the happy path: schema conformance for every tool, correct behavior on invalid and missing parameters, error responses the model can actually recover from, and stateful sequences where one call feeds the next. An MCP server that returns opaque errors will produce agents that hallucinate success.
The highest-value tests are the ones that inject failure. Return a 500, a rate limit, a not-found — then assert the agent retries with corrected parameters or changes approach rather than fabricating a result. You cannot discover recovery behavior without deliberately breaking things.
On every change that can alter model behavior: prompt edits, model version bumps, retrieval or tool changes, and dependency upgrades. The practical pattern is a fast subset on every pull request with a full suite nightly, gated on a baseline with an explicit tolerance band rather than exact-match scores.
Evals that run once are a report; evals that run on every change are a quality gate. Because model outputs vary between runs, the gate needs a tolerance band and a pinned configuration — fixed seeds and temperatures, versioned prompts — or it will fail randomly and be disabled within a week.
Constraint stacking, negation, temporal reasoning, format compliance under pressure, and long-output self-consistency. Systematic adversarial testing across GPT, Claude, Gemini and Hunyuan classes surfaces these before users do — and each confirmed break becomes a regression test with pinned seeds, temperatures, and versioned prompts.
Every model has a breaking surface, and it moves with each release. That is why finding a failure is only half the work: without capturing it as a pinned regression test, the same failure silently returns two versions later.