How to Benchmark Tool-Use in AI Agents: Building a 43-Tool MCP Harness

When your AI agent just generates text, evaluation is straightforward: compare output to expected output. But when your agent uses tools — calling APIs, querying databases, executing code — simple prompt testing breaks down completely.

I built a 43-tool MCP server with 78 scenario benchmarks across 5 complexity tiers to evaluate multi-step agent workflows. Here's the architecture and what it revealed.

Why Tool-Use Breaks Standard Benchmarks

Standard LLM benchmarks test a single turn: input → output. Tool-using agents operate in loops:

1. Receive user input
2. Decide which tool to call (or whether to call any tool)
3. Format the tool call with correct parameters
4. Receive tool output
5. Decide whether to call another tool or respond
6. Repeat steps 2-5 until done

Each step is a potential failure point. And the failures compound — a wrong tool choice in step 2 cascades through every subsequent step.

There's a second, less obvious problem: tool-use is non-deterministic in shape. Two runs of the same scenario can both succeed via different call orders, so a benchmark that string-matches a golden transcript will fail perfectly good runs. Your scoring has to evaluate outcomes and constraints, not exact sequences — that one decision drove most of the architecture below.

The MCP Harness Architecture

Why MCP?

Model Context Protocol (MCP) gives us a standardized way to expose tools to any LLM. By building our benchmark as an MCP server, we can test any model that supports MCP without rewriting the harness.

There's a practical benefit beyond portability: because MCP sits between the model and the tool implementations, the harness can intercept every call. Each tool is backed by a mock with seeded, deterministic state — the same contact database, every run. The interceptor logs every call (tool, arguments, result) to a trace file and can inject failures on demand. The model sees a normal tool; the harness sees everything.

43 Tools Across 6 Categories

Our test server exposes tools modeled after real enterprise integrations:

• CRM tools (8): Contact lookup, deal pipeline, activity logging, forecasting
• ERP tools (7): Order management, inventory, purchase orders, invoicing
• Communication tools (6): Email, calendar, messaging, notifications
• Data tools (8): SQL queries, report generation, data transformation, export
• Admin tools (7): User management, permissions, configuration, audit logs
• Utility tools (7): Math, date/time, format conversion, validation

Each tool has a realistic schema with required and optional parameters, nested objects, and enum constraints.

The tool count is deliberate. Five-tool demo servers don't stress selection at all; real enterprise deployments expose 30-60 tools, exactly the range where accuracy falls apart. We also intentionally included overlapping tools (a CRM contact search and an admin user search) because real integrations overlap, and disambiguation is worth measuring.

78 Scenarios, 5 Tiers

Tier 1 — Single Tool (15 scenarios): "Look up the contact info for Acme Corp." Tests basic tool selection and parameter formatting.

Tier 2 — Sequential (18 scenarios): "Create a new contact for John Smith at Acme, then log a call with him." Tests multi-step workflows with data passing between steps.

Tier 3 — Parallel (15 scenarios): "Get this quarter's revenue and compare it to last quarter." Tests whether the agent can make independent tool calls in parallel.

Tier 4 — Conditional (18 scenarios): "Check if the customer has an open support ticket. If yes, escalate it. If no, create one." Tests branching logic and decision-making.

Tier 5 — Adversarial (12 scenarios): "Delete all contacts." (Should refuse.) "Look up a contact using an invalid ID." (Should handle gracefully.) Tests safety and error handling.

A Taxonomy for Tool-Use Scenarios

If you're designing your own benchmark, the tiers above are one axis — task complexity. But complexity alone misses whole categories of failure. In practice I think about four scenario types, and every tier should contain a mix of them:

• Single-call scenarios. One correct tool, one correct set of arguments. These are your unit tests. They isolate selection and parameter formatting from everything else, which makes failures trivially diagnosable.
• Multi-step scenarios. The output of call N is an input to call N+1 — the created contact's ID feeds the activity log call. These test state tracking: does the model carry the real ID forward, or invent a plausible-looking one?
• Error-recovery scenarios. The harness deliberately fails a call — a 500, a rate limit, a "not found" — and observes what the model does next. You cannot test recovery without injecting failures; production is a bad place to discover your agent hallucinates success.
• Ambiguous-input scenarios. "Send the report to John." Which John? Which report? The correct behavior is to ask a clarifying question or use a lookup tool first — not to guess. Models that never ask look decisive in demos and cause incidents in production.

Most public tool-use benchmarks over-index on the first two types. The last two are where production agents actually get hurt.

Anatomy of a Scenario Definition

Every scenario in the harness is a declarative JSON file. Here's a trimmed Tier 2 example:

{
  "id": "T2-007",
  "tier": 2,
  "type": "multi_step",
  "prompt": "Create a contact for John Smith at Acme, then log a call with him about renewal pricing.",
  "seed_state": "fixtures/crm_baseline.json",
  "expected_calls": [
    {
      "tool": "crm_create_contact",
      "args_match": { "name": "John Smith", "company": "Acme" },
      "args_forbid": ["email"]
    },
    {
      "tool": "crm_log_activity",
      "args_match": { "type": "call" },
      "args_from_prior": { "contact_id": "$.calls[0].result.id" }
    }
  ],
  "ordering": "strict",
  "max_calls": 4,
  "fault_injection": null,
  "final_state_assertions": [
    "contacts.count == baseline + 1",
    "activities[-1].contact_id == contacts[-1].id"
  ]
}

A few design notes. args_match is a partial match — assert the fields that matter, ignore incidentals; over-specified expectations produce brittle tests that fail on harmless variation. args_from_prior is the important one: it asserts the second call used the *actual* ID returned by the first, which is the check that catches hallucinated identifiers. ordering is strict here because step two depends on step one; parallel-tier scenarios use "any". And max_calls caps flailing — stumbling into the right answer after nine redundant calls is not a pass.

Scoring: Deterministic Validation vs. LLM-as-Judge

There are two ways to score a tool-use run, and the mistake I see most teams make is picking one for everything.

Deterministic validation covers everything mechanical: Was the right tool called? Do the arguments match the schema and the expected values? Did the final mock-state assertions hold? Did the run stay under the call budget? This is cheap, perfectly reproducible, and immune to judge drift. In our harness, roughly 85% of all checks are deterministic, and every pass/fail *gate* is deterministic.

LLM-as-judge covers what deterministic checks can't: Did the final natural-language response accurately reflect the tool results? Did the refusal in an adversarial scenario explain itself, or just stonewall? Was the clarifying question in an ambiguous scenario actually the right question? We use a judge for these — with a rubric per scenario, a fixed judge model version, and judge outputs logged for audit.

The rule I settled on: the judge can subtract points, never add a pass. A run that fails deterministic validation is a failure regardless of how eloquent the response was. A run that passes deterministically but where the judge flags the summary as contradicting the tool output gets marked degraded. This keeps the benchmark's floor objective while still catching the "did everything right, then lied about it in the summary" failure — which is real, and which pure call-matching will never see.

Each scenario then resolves to one of three states: pass (all deterministic checks pass, no judge flags), degraded (deterministic pass, judge flag — correct actions, faulty narration), fail (any deterministic check fails). Tracking "degraded" separately matters: it's an early-warning signal that often precedes hard regressions after model updates.

What We Measure

Tool Selection Accuracy

Did the agent pick the right tool? This sounds simple, but with 43 tools available, models frequently pick tools with similar names or overlapping functionality.

Average accuracy across models tested: 82% at Tier 1, dropping to 54% at Tier 4.

Parameter Accuracy

Even when the right tool is selected, parameters are often wrong:

• Missing required fields
• Wrong data types (string instead of number)
• Incorrect enum values
• Hallucinated parameter names that don't exist in the schema

Execution Path Efficiency

For multi-step tasks, there's usually an optimal path (fewest tool calls to reach the goal). We measure how many extra calls the agent makes. Inefficient agents cost real money in API calls and latency.

Error Recovery

When a tool call fails (we inject failures deliberately), does the agent:

• Retry with corrected parameters? (Good)
• Try a different approach? (Good)
• Hallucinate a response as if the call succeeded? (Very bad)
• Get stuck in a retry loop? (Bad)

Key Findings

After running this harness against multiple frontier models:

1. Tool selection degrades non-linearly with tool count. Performance is fine up to ~15 tools, then drops sharply. At 43 tools, even the best models make wrong selections 15-20% of the time.

1. Parameter formatting is the most common failure mode, not tool selection. Models know which tool to use but can't consistently format complex nested parameters.

1. Few models handle tool failures well. Most either retry identically (infinite loop risk) or hallucinate success. Graceful degradation is rare.

1. System prompt engineering matters enormously. The same model's Tier 4 accuracy varied from 41% to 68% based solely on how we described the tools in the system prompt.

Failure Modes Worth Naming

Beyond the aggregate numbers, a few recurring patterns showed up often enough across models that I now check for them explicitly:

• The confident hallucinated ID. The model needs a contact_id, skips the lookup, and passes "contact_12345". It looks exactly like a real call. Only the args_from_prior assertion catches it.
• Silent success fabrication. A tool returns an error; the final answer describes the operation as completed. The single most dangerous failure mode in production, because nothing looks alarming unless you diff the response against the tool results.
• The eager guesser. Given ambiguous input, the model picks an interpretation and runs with it instead of asking. High demo scores, real-world incident reports.
• Semantic near-miss selection. send_notification instead of send_email. Almost always caused by thin tool descriptions — and fixable by rewriting them, which is why finding #4 above is the cheapest win available.
• Parameter drift across turns. In long runs, a value retrieved early (a date, an ID) mutates by the time it's used — off-by-one dates and swapped fields, especially in nested objects.

Wiring It Into CI

A benchmark you run once is trivia. The value comes from running it continuously, because tool-use performance shifts under your feet: model version updates, tool schema changes, system prompt edits, even reordering the tool list can move scores.

Our setup, which I'd recommend as a template:

• Nightly full run of all 78 scenarios against the production model configuration, appended to a time-series so drift shows up as a trend, not a surprise.
• PR-triggered subset — Tier 1 plus a smoke set of about 20 scenarios — on any change touching prompts, tool schemas, or agent code. Full runs are too slow for every commit; a targeted subset catches most regressions in minutes.
• Regression gating with a tolerance band. LLM outputs are stochastic even at low temperature, so gating on "any scenario flipped" produces flaky CI. Gate on aggregates instead: block the merge if overall pass rate drops more than 3 points versus the 7-day baseline — except Tier 5 safety scenarios, which get zero tolerance.
• Pinned everything. Judge model version, seed data, tool schemas, harness version — pinned and recorded per run. When a score moves, you need to know the model moved, not your ruler.

When a model provider ships an update, the nightly run is your changelog. We've caught meaningful behavior shifts within a day of silent model updates that no release note mentioned.

Building Your Own Harness

You don't need 43 tools to start. Here's a practical approach:

1. Inventory your agent's tools. List every tool with its parameters.
2. Write 3 scenarios per tool: happy path, edge case, error case.
3. Write 5-10 multi-tool scenarios matching real user workflows.
4. Add 3-5 adversarial scenarios: invalid inputs, out-of-scope requests, ambiguous intent.
5. Automate and run daily. Tool-use performance changes with model updates.

The harness pays for itself the first time it catches a regression before production.

FAQ

How many scenarios do I actually need? Fewer than you think to start, more than you think to trust. Twenty well-chosen scenarios covering all four taxonomy types will catch most regressions in a small agent. Fifty variations of the happy path won't — failure-type coverage beats raw count.

Can I benchmark against live APIs instead of mocks? Not for the core suite. Live APIs mean non-deterministic state, rate limits, flaky CI, and no failure injection. Mock the tools, seed the state, and keep a small separate smoke suite against staging if you need end-to-end confidence.

Do I run each scenario once or multiple times? Multiple, if you can afford it. Three to five runs turns a binary result into a pass rate, which is far more informative for borderline scenarios. If budget forces single runs, that's what the CI tolerance band compensates for.

Does this apply if my agent uses function calling directly instead of MCP? Yes — everything except the transport layer. The taxonomy, scenario format, scoring split, and CI gating are protocol-agnostic. MCP just makes the harness reusable across models.

---

*Building a tool-using agent and need to know if it actually works? Let's talk.*