How to Evaluate an AI Agent Before Launch: A Practical Framework

Most teams test AI agents the same way they test chatbots: type a few prompts, eyeball the output, ship it. Then production users find every edge case you missed.

I've evaluated agents for enterprise clients across SAP integrations, customer support, and code generation. The pattern is always the same: demo performance and real-world reliability are completely different things.

An agent that looks flawless in a demo can fail a third of realistic multi-step tasks. Not because the model is bad — because nobody measured. This is the framework I use to measure before launch.

The Problem with Ad-Hoc Testing

When you test an agent with 10-20 hand-picked prompts, you're testing the happy path. You're confirming what you already believe works. You're not finding:

• Multi-step failures where step 3 breaks because step 1 returned an unexpected format
• Tool selection errors where the agent picks the wrong API for a valid request
• Graceful degradation failures where the agent hallucinates instead of saying "I don't know"
• Context window overflow behavior on long conversations
• Concurrent tool-call race conditions

Each of these hides from demo testing for a structural reason.

Multi-step failures compound. If each step succeeds 95% of the time, a six-step workflow succeeds roughly 74% of the time. Demos rarely go past step two, so you never see the compounding.

Tool selection errors hide behind plausible output. The agent calls the wrong endpoint, gets a real response, and writes a confident summary of the wrong data. Unless your test asserts *which* tool was called with *which* arguments, this passes eyeball review every time.

Graceful degradation is the failure that hurts most in production. Users forgive "I can't find that order." They don't forgive a fabricated tracking number. You only learn which one your agent produces by deliberately asking questions it cannot answer — which nobody does in a demo.

Context and concurrency failures only appear under load. A 40-turn conversation. Three tool calls in flight at once. No demo reaches those conditions by accident; you have to construct them on purpose.

What a Good Eval Scenario Looks Like

Before the framework, get the unit of work right. A scenario is not a prompt — it's a prompt plus an explicit definition of success. I write mine as structured cases the harness can execute and score:

{
  "id": "T3-07",
  "tier": 3,
  "prompt": "Customer arjun.k@example.test says his last order never arrived. Check the order status and if it shipped more than 10 days ago, open a lost-package case.",
  "setup": {
    "seed_data": "orders_fixture_v2.json",
    "mocked_tools": { "carrier_api": { "status": "in_transit", "shipped_days_ago": 14 } }
  },
  "expected": {
    "tools_called": ["lookup_customer", "get_order_status", "create_case"],
    "forbidden_tools": ["issue_refund"],
    "must_assert": ["case created with correct order_id", "no invented tracking events"],
    "final_state": "case exists in CRM fixture"
  }
}

Three details matter here:

1. Seeded, deterministic data. Every scenario runs against a fixture, not a live system. If the data drifts, your scores mean nothing week over week.
2. Assertions on the trajectory, not just the answer. tools_called and forbidden_tools catch the agent that reaches a right-looking answer the wrong way — or does something destructive along the way.
3. A final-state check. For agents that write data, the database is the ground truth. The chat transcript can look perfect while the record is wrong.

The 78-Scenario Framework

After building evaluation harnesses for multiple clients, I've converged on a framework that covers 78 multi-step scenarios across 5 complexity tiers:

Tier 1: Single-Tool, Single-Step (15 scenarios)

Basic sanity checks. Can the agent call the right tool with the right parameters? These should have a 95%+ pass rate before you move on.

Cover every tool at least once, with clean inputs. "Look up customer by email." "Get the status of order 4417." If the agent formats a date wrong or passes a name where an ID belongs, you want to know here, where the failure is cheap to diagnose — not buried in a five-step workflow.

Tier 2: Single-Tool, Multi-Step (18 scenarios)

Sequential operations using one tool. Create → Read → Update → Delete flows. Tests whether the agent maintains state across steps.

The classic failure: the agent creates a record, receives an ID in the response, then ignores that ID and hallucinates one for the update call. Your assertions should check that outputs from step N actually flow into step N+1.

Tier 3: Multi-Tool, Guided (15 scenarios)

The user's intent requires multiple tools, but the path is relatively clear. "Look up this customer's order and update their shipping address."

This is the first tier that resembles real usage, and it's where most agents show their first meaningful drop. Watch for two patterns: skipped lookups (the agent updates without reading first, guessing at fields) and redundant calls (fetching the same record three times, which burns latency and tokens).

Tier 4: Multi-Tool, Ambiguous (18 scenarios)

The user's request could be handled multiple ways. Tests whether the agent picks a reasonable path and asks for clarification when needed.

Scoring here needs judgment: several trajectories can be correct. I define an *acceptable set* of paths per scenario, plus explicit tripwires — "the customer has two open orders and the prompt doesn't say which; the agent must ask, not pick one." An agent that guesses confidently on ambiguous input is a bigger risk than one that fails loudly.

Tier 5: Adversarial and Edge Cases (12 scenarios)

Invalid inputs, contradictory instructions, context window limits, tool failures. This is where most agents fall apart.

My standard set: tools returning 500s and timeouts mid-workflow, empty result sets the agent must not paper over, instructions that contradict system policy ("ignore the refund limit"), prompt-injection strings embedded in tool output, and inputs designed to overflow context. The pass criterion is rarely "completed the task" — it's "failed safely, told the truth, left no bad writes behind."

Scoring Beyond Pass/Fail

Binary pass/fail misses the nuance. I score each scenario on:

• Correctness (0-3): Did the agent produce the right output?
• Efficiency (0-3): Did it take a reasonable path, or did it make unnecessary tool calls?
• Graceful failure (0-3): When it couldn't complete the task, did it fail helpfully?

A score of 7+ out of 9 is "production ready." Below 5 needs significant work.

To keep scoring consistent across runs (and across reviewers), each dimension has anchored levels:

Two rules keep the numbers trustworthy. First, any destructive wrong write scores 0 on correctness, however good the transcript reads. Second, run each scenario 3-5 times — agents are stochastic, and a scenario that passes 2 of 5 runs is a fail, not a "flaky test." Track pass rate per scenario, not the best run.

For grading free-text outputs at scale I use an LLM judge with the rubric above pasted into its prompt — but I hand-grade a random 10% every cycle to check the judge's agreement rate. If judge and human disagree more than ~5% of the time, the rubric is too vague, and the fix is sharper anchors, not a smarter judge.

A Failure-Mode Taxonomy

Scores tell you *how bad*; tagging tells you *what to fix*. Every failed run gets one primary tag:

After a full run, sort by tag frequency. In my experience the top two tags account for well over half of failures, and they're almost always fixable with prompt and tool-schema changes — no fine-tuning, no model swap. That's the entire point of running evals before launch: you fix categories, not one-off bugs.

Building the Harness

You don't need a platform. My harnesses are usually a few hundred lines:

for scenario in load_scenarios():
    reset_fixtures(scenario.setup)
    for run in range(5):
        transcript = run_agent(scenario.prompt, mocks=scenario.setup.mocked_tools)
        result = {
            "trajectory": check_tools(transcript, scenario.expected),
            "state": check_final_state(scenario.expected.final_state),
            "rubric": judge(transcript, scenario.expected.must_assert),
        }
        log(scenario.id, run, result)
report()  # pass rate per tier, score distribution, failure tags

The non-negotiables: fixture reset between runs, full transcript capture (every tool call and result, not just the final answer), and versioned scenario files in git. When you change a prompt, you re-run the suite and diff pass rates against the last commit. That turns "I think the new prompt is better" into a number — and catches the regression where fixing Tier 4 quietly broke Tier 2.

What This Looks Like in Practice

For a recent SAP integration agent, this framework revealed that the agent:

• Passed 95% of Tier 1-2 scenarios
• Dropped to 67% at Tier 3
• Failed 80% of Tier 5 adversarial cases

The client had been testing with Tier 1 prompts only and thought they were ready to launch. The Tier 3-5 failures would have hit production users within the first week.

The tags told the story: most Tier 3 failures were *lost state* — the agent re-queried instead of reusing results, and occasionally acted on the stale copy. Most Tier 5 failures were *silent failure* — a mocked API timeout, and the agent reported success anyway. Both were fixed with prompt and error-handling changes in days, and the re-run moved Tier 3 into the 90s. Without the harness, those same failures would have surfaced one support ticket at a time.

Getting Started

You don't need 78 scenarios on day one. Start with:

1. Map every tool your agent can call
2. Write 3 scenarios per tool (happy path, edge case, failure case)
3. Write 5 multi-tool scenarios that match real user workflows
4. Run them systematically, score them, track them over time

For an agent with five tools, that's twenty scenarios — a focused day of work that catches most of what a full harness catches. Then grow the suite from production: every real failure becomes a new scenario, so the same bug can never ship twice.

Before you launch, I hold the bar at:

• Tier 1-2 at 95%+ over repeated runs
• Tier 3 at 85%+, Tier 4 at 75%+
• Zero destructive wrong writes anywhere in the suite
• Every Tier 5 failure reviewed and classified as safe-to-ship or blocking

The goal isn't perfection — it's knowing where you stand before your users find out for you.

FAQ

How many scenarios do I actually need? Enough to cover every tool, every realistic workflow, and every failure mode you can name — for most agents that's 20 at the start and 60-100 at maturity. Coverage matters more than count: 30 scenarios spread across all five tiers beat 100 happy-path prompts.

Can I use an LLM to judge the outputs? Yes, and at scale you'll have to — but keep trajectory and state checks as plain code (a tool was called or it wasn't), reserve the LLM judge for free-text quality, give it an anchored rubric, and audit a sample of its grades by hand every cycle.

How often should I re-run the suite? On every prompt change, tool change, and model version bump — the suite is your regression net, so wire it into CI if you can. A model upgrade is not automatically an improvement for *your* agent, and the diff between two runs is how you find out.

What pass rate is good enough to launch? It depends on blast radius. A read-only research agent can ship at Tier 3 ~80% if it fails honestly. An agent that writes to an ERP or moves money needs 95%+ on its core tiers and a clean Tier 5 — because there the cost of one bad run isn't a wrong answer, it's a wrong record.

---

*Want help building an evaluation harness for your AI agent? Book a free 20-minute call and I'll tell you honestly whether I can help.*