Building an Eval Suite for an Autonomous Agent (Without Fooling Yourself)

Table of Contents

• Why Agent Evals Are a Different Beast
• Start With the Failure Modes, Not the Tests
• Defining "Success" Before You Measure It
• Building the Golden Dataset
• Choosing Your Graders
• Deterministic Checks
• Rubric and LLM-as-Judge Grading
• Trajectory vs. Outcome Scoring
• Wiring It Into the Development Loop
• The Metrics That Actually Matter
• Insights Most People Overlook
• References

Why Agent Evals Are a Different Beast

If you've shipped software before, you have instincts about testing. Most of those instincts will mislead you here.

A traditional unit test is deterministic: the same input produces the same output, and a diff tells you instantly whether something broke. An autonomous agent breaks every assumption in that sentence. The same prompt can produce a different trajectory on two consecutive runs, different tool calls, different intermediate reasoning, sometimes a different final answer that's also correct. The agent might take twelve steps one time and four the next. It might reach the right answer through a path that would horrify you if you read the trace.

This is the core problem that makes agent evaluation its own discipline inside the GaaS world, and it's tightly bound up with the reproducibility problem, the maddening reality that identical inputs yield non-identical runs. You aren't testing a function. You're testing a stochastic decision-maker that has agency over its own steps.

The stakes are also asymmetric. When you sell an agent on a per-task or per-outcome basis, a failure isn't a stack trace a developer sees in staging, it's a customer's refund processed twice, a sales email sent to the wrong segment, a contract clause misread. For most GaaS companies the eval suite isn't a nice-to-have engineering hygiene item. It's the closest thing they have to a warranty. That's why a growing number of vendors are putting a reliability number on the homepage and treating it as a sales asset, not an internal metric.

Start With the Failure Modes, Not the Tests

The most common mistake teams make is starting with "what should the agent do?" and writing happy-path cases. That gives you a suite that passes confidently and tells you nothing.

Flip it. Start with: how does this agent fail? Sit with your support tickets, your trace logs, your angriest customer emails. You'll find the failures cluster into recognizable shapes:

• Silent failures, the agent confidently returns something plausible and useless. It "completed" the task by doing nothing of value. This is the most dangerous category because nothing throws an error.
• Tool-call failures, it calls the right tool with malformed arguments, or the wrong tool entirely, or loops calling the same tool.
• Right answer, wrong reason, it lands the correct output through reasoning that won't generalize, so it'll fail the next slightly different case.
• Did the thing correctly, but it was the wrong thing, perfect execution of a misunderstood intent.
• Hallucinated grounding, it invents a fact, a record ID, an account balance.

Each of these deserves its own slice of your eval suite. If you only have one number, "task success rate", you can't tell which of these is dragging it down, and you can't tell whether last week's fix for tool-call errors quietly made hallucinations worse. Anthropic's own guidance on building evaluations for agentic systems makes this point repeatedly: granular, failure-mode-aligned evals beat a single aggregate score every time.

Write your failure taxonomy down. It becomes the skeleton of everything that follows.

Defining "Success" Before You Measure It

Here's a question that sounds trivial and isn't: when your agent books a meeting, did it succeed?

Did it find a real open slot? Did it pick a time the user would actually want? Did it send a confirmation? Did it avoid double-booking? Did it use the right calendar? "Success" turns out to be a small bundle of conditions, and if you don't enumerate them explicitly, your grader will quietly reward partial credit you never intended to give.

For each task type, write a success contract, a plain-language definition of done, decomposed into checkable conditions. Some conditions are binary (a confirmation email was sent: yes/no). Some are graded (the proposed time was reasonable: 0-3). The discipline of writing this down forces arguments out into the open before they show up as a customer dispute over whether you owe a refund on a per-outcome contract.

A useful frame from the broader testing world: the distinction between verification (did the agent do the thing right?) and validation (did it do the right thing?). Most weak eval suites only check verification. The expensive failures live in validation, the agent that flawlessly executed the wrong interpretation of what the user meant.

Building the Golden Dataset

Your eval is only as good as the cases you feed it. The golden dataset, a curated set of inputs paired with known-good expected outcomes, is the asset that takes the longest to build and that competitors find hardest to copy. It's worth doing well.

A few hard-won principles:

Mine production, don't invent. Synthetic cases are fine for bootstrapping and for stress-testing edge conditions you can't wait for, but the spine of your dataset should be real traffic, anonymized, with real messiness. Users phrase things in ways you'd never write. The mix of synthetic and real cases is a genuine tradeoff worth tuning deliberately rather than defaulting to whatever's easy.

Oversample the long tail. Random sampling of production traffic gives you a dataset that's 80% easy cases, which means your headline score is dominated by problems you already solved. Deliberately overweight the weird, the ambiguous, the multi-step, the cases near your agent's competence boundary. That's where regressions hide.

Capture the full context, not just the prompt. For agents with memory or state, the same prompt means different things depending on what came before. Your golden case needs to pin down the state the agent starts from, or you can't reproduce it.

Version it and review it like code. Expected outputs drift. A case that was "correct" six months ago may now reflect a stale business rule. Treat the golden dataset as a living artifact with owners and review, this is exactly the kind of work that's pushing GaaS companies to staff a dedicated eval team rather than leaving it to whoever has spare time.

Budget realistically: a serious vertical agent needs hundreds of well-curated cases, not dozens, and the curation is human-intensive. McKinsey's analysis of what separates AI leaders in moving agents into production keeps returning to the same theme, the differentiator is rarely the model, it's the surrounding evaluation and data discipline.

Choosing Your Graders

Once you have cases, you need to automatically decide whether the agent passed. This is where most of the engineering subtlety lives.

Deterministic Checks

Wherever you possibly can, grade with code. If the success contract says "a refund was issued for $42.50," check the database. If it says "the output is valid JSON matching this schema," parse it. Deterministic graders are fast, free, perfectly reproducible, and never have an opinion. The trap is forcing a deterministic grader onto an inherently fuzzy judgment, exact-string-matching a free-text reply will fail every time the agent rephrases correctly. Reserve code graders for conditions that genuinely have a crisp answer.

Rubric and LLM-as-Judge Grading

For the fuzzy conditions, was the tone appropriate, was the summary faithful, was the recommendation sensible, you'll reach for a model to grade the output against a rubric. LLM-as-judge is powerful and now standard, but it comes with sharp edges:

• Judges are biased. They favor longer answers, their own phrasing style, and outputs that sound confident. Calibrate the judge against human-labeled examples before you trust its numbers.
• A judge is itself an agent that can fail silently. If your grader hallucinates, your "95% pass rate" is fiction. Spot-check the judge's verdicts against human review on a sample, continuously.
• Give the judge a rubric, not a vibe. "Rate 1-5" produces noise. "Score 0 if any fact is unsupported by the source; score 3 only if every claim is grounded" produces signal.

The pragmatic move is a hybrid: deterministic checks for everything verifiable, a calibrated LLM judge for the rest, and a small slice routed to humans to keep both honest. Human review workflows are their own design problem once volume scales, but a thin, continuous human-in-the-loop sample is what keeps the automated graders from drifting into comfortable fiction.

Trajectory vs. Outcome Scoring

Do you grade what the agent did (the trajectory, which tools, in what order) or what it produced (the outcome)? Both, but weight outcome heavily. Over-fitting to a "correct" trajectory punishes the agent for finding a better path and bakes today's approach into your tests. Grade trajectory for the things that genuinely matter, did it avoid a destructive action, did it stay within its permission scope, and grade outcome for whether the job got done. End-to-end trace inspection still matters enormously for debugging a failure; it just shouldn't be the primary pass/fail signal.

Wiring It Into the Development Loop

An eval suite that runs once before launch is a benchmark, not an eval suite. The value compounds only when it runs constantly.

Treat evals the way mature teams treat tests: part of the development inner loop. The emerging practice, call it eval-driven development, is to write or update the eval before changing the prompt or the tool definitions, so you're moving a number you can see rather than eyeballing a few examples and declaring victory. Engineers describe this shift the way they once described test-driven development: slower to start, dramatically faster once the harness exists.

Concretely, you want:

• Pre-merge gates. Run a fast subset of the suite on every change to the agent's prompts, tools, or orchestration. Block the merge if the success rate on critical slices regresses.
• Regression coverage for model swaps. When the model underneath you changes, a provider upgrade, a version deprecation, a switch to a cheaper model, rerun the full suite. This is the single most underrated use of an eval suite, because a model change can silently shift behavior across thousands of cases with no code diff at all.
• Shadow and canary stages. Before a change reaches all traffic, run the new agent in shadow mode against live inputs and compare against the current one, or roll it out as a canary to a slice of users. Your eval suite is what makes those comparisons meaningful instead of anecdotal.
• Continuous production evals. Pre-launch evals tell you the agent was good on the day you tested. Sampling live traffic and scoring it continuously tells you whether it's still good, and it's how you catch slow drift before a customer does.

The Metrics That Actually Matter

Resist the urge to collapse everything into one number, and be deeply suspicious of impressive-sounding accuracy figures. "99% accurate" is close to meaningless for an agent unless you know: 99% of what, on which distribution of cases, scored by whom. A 99% that's measured on easy cases and a 92% measured on the hard tail are not comparable, and the 92% is probably the more honest number.

Track at minimum:

• Task success rate, sliced by failure mode and by difficulty. The aggregate hides everything; the slices are where decisions live.
• The false-positive vs. false-negative split. In some verticals a wrong action is catastrophic and a missed action is merely annoying; in others it's reversed. Weight your scoring to match the real-world cost, because they are almost never equal.
• Cost and latency per task, alongside quality. An agent that's marginally more accurate but twice as slow or twice as expensive may be a worse product. The latency-reliability tradeoff is real and belongs in your eval dashboard, not in a separate spreadsheet.
• Trend over time, not just the current value. A flat 94% that's been slowly sliding from 97% is an emergency disguised as a passing grade.

The goal of all of this isn't a perfect score. It's an honest one, a number you'd be comfortable putting in front of an enterprise buyer who's about to bet a workflow on you.

Insights Most People Overlook

Your eval suite is a moat, not a chore. Capability is increasingly commoditized, the frontier models are available to your competitors too. What they can't easily copy is your accumulated golden dataset, your failure taxonomy refined over thousands of real incidents, and a grading harness calibrated against your specific domain. In the GaaS market, the reliability infrastructure is harder to replicate than the agent itself. Teams that treat evals as overhead are giving away the one thing that compounds.

The grader is the most dangerous unmonitored agent you run. Everyone scrutinizes the production agent. Almost nobody continuously audits the LLM-as-judge that decides whether the production agent passed. A drifting or biased judge doesn't just give you wrong numbers, it gives you confidently wrong numbers, which is worse than no numbers, because it manufactures false safety. Eval the evaluator.

A passing eval suite that never fails is broken. If your suite is green every single run, your cases are too easy and clustered on solved problems. A healthy suite lives near the edge of the agent's competence and fails sometimes, that's the signal you're actually measuring the boundary where regressions happen, not re-confirming what already works.

Per-outcome pricing turns eval gaps into direct revenue leakage. When you bill per successful outcome, every false positive in your grader, every case your suite calls "success" that a customer would dispute, is money you'll refund or trust you'll lose. The economic model of GaaS makes grader precision a P&L line item, not an engineering nicety. The teams that internalize this build their success contracts to match the contract they signed with the customer, literally.

Synthetic data is great for breadth and terrible for calibration. Generated cases let you probe edge conditions cheaply, but they share the blind spots of whatever model generated them, and they systematically miss the texture of real user weirdness. Use synthetics to widen coverage; never let them set your headline reliability number. The number you quote should come from real traffic.

References

• Anthropic, Develop tests and evaluations for Claude applications
• McKinsey & Company, QuantumBlack insights on AI in production