Continuous Evaluation in Production: Why Pre-Launch Testing Is the Easy Part

Table of Contents

• The Pre-Launch Trap
• Why Agents Decay After You Ship
• What "Continuous Evaluation" Actually Means
• The Three Layers of Online Evaluation
• Layer 1: Cheap Heuristics on Every Run
• Layer 2: LLM-as-Judge on a Sample
• Layer 3: Human Review on the Hard Cases
• Sampling Without Going Broke
• Closing the Loop: From Signal to Fix
• What This Means for GaaS Pricing and Trust
• Insights Most People Overlook
• References

The Pre-Launch Trap

There's a comfortable lie that software teams have told themselves for forty years: test it thoroughly, ship it, move on. For deterministic code, that lie was mostly harmless. The function that added two numbers in March still added them in June. Your test suite was a fence, and once the fence was up, the field stayed fenced.

Agents broke the fence.

The thing that makes Agentic AI-as-a-Service valuable, that it reasons, plans, and acts across multi-step workflows instead of returning a fixed output, is exactly the thing that makes a one-time eval worthless. You are not shipping a function. You are shipping a probabilistic system that sits on top of a foundation model you don't control, consumes inputs you can't predict, and produces a distribution of behaviors rather than a single answer. A pre-launch eval captures one snapshot of that distribution at one moment. It tells you almost nothing about the next moment.

I've watched teams celebrate a 92% task-success score in their golden dataset, deploy on Friday, and spend the following Wednesday confused about why support tickets tripled. The eval wasn't wrong. It was just stale by the time anyone read it. Pre-launch evaluation answers "is this good enough to turn on?" Continuous evaluation answers the question that actually keeps customers: "is this still good enough right now?"

Why Agents Decay After You Ship

Decay is not a metaphor here. It's a measurable, repeatable phenomenon, and it comes from at least four independent sources that pre-launch testing structurally cannot catch.

The model moves underneath you. If you're building on a hosted foundation model, the provider updates weights, adjusts system behavior, deprecates a checkpoint, or silently reroutes you to a new version. Your prompt didn't change; the thing interpreting it did. This is the single most underrated risk in GaaS, and it's why regression testing when the model underneath changes has become its own discipline. An agent tuned against one model version can behave meaningfully differently the morning after a provider rolls an update.

Input distribution drifts. The queries your agent saw in your test set were the queries you imagined. Real users ask things you didn't. As your product finds new customer segments, the input distribution moves away from your golden dataset. Three months in, a sizable share of production traffic may look nothing like what you evaluated against.

The world changes. A vertical agent that books travel, files compliance paperwork, or reconciles invoices is coupled to external reality, APIs change schemas, regulations update, prices move, a downstream tool starts returning a new error format. Your agent's logic is frozen; its environment isn't.

Compounding error in long horizons. In multi-step workflows, a small per-step error rate compounds. A 2% chance of a wrong tool call per step sounds fine until you string twelve steps together and watch end-to-end success quietly erode. Stanford's AI Index report has repeatedly noted the gap between benchmark performance on isolated tasks and reliability across realistic, multi-stage workflows, and that gap is precisely where production decay lives.

None of these are bugs you can fix once. They are ongoing pressures. Which means the response can't be a one-time fix either.

What "Continuous Evaluation" Actually Means

Let's be precise, because the term gets thrown around loosely. Continuous evaluation (often called online evals, as opposed to offline evals run against a static dataset) is the practice of scoring your agent's real production behavior, continuously, against quality criteria, and routing those scores into alerts, dashboards, and improvement loops.

The distinction from observability matters. Plenty of teams have logging and tracing, they can replay an agent run end to end and see every tool call. That's necessary but not sufficient. Observability tells you what happened. Continuous evaluation tells you whether what happened was good. A trace shows you the agent called the refund API. An eval tells you it should have escalated to a human instead. As a16z has argued in its writing on the emerging AI infrastructure stack, the tooling layer for judging agent quality in production is forming into its own category precisely because traces alone don't answer the quality question.

Three properties define a real continuous-eval system:

1. It runs on live traffic, not just synthetic test cases.
2. It produces a quality signal, not just a log, a score, a pass/fail, a flag.
3. It feeds a loop, alerting, triage, and ideally automated regression-suite growth.

If you have all three, you have continuous evaluation. If you're missing the third, you have a very expensive dashboard nobody looks at.

The Three Layers of Online Evaluation

You cannot run your most rigorous, expensive eval on every single production request, the cost and latency would be absurd. The teams that get this right run a tiered system: cheap checks everywhere, expensive checks selectively. Think of it as a funnel.

Layer 1: Cheap Heuristics on Every Run

Run on 100% of traffic, cost effectively zero, latency negligible. These are deterministic, code-based checks that don't need a model to compute:

• Did the agent produce valid, schema-conformant output?
• Did it complete within the step budget, or did it loop?
• Did every tool call return successfully, or did something throw?
• Are there obvious red flags, empty responses, refusals, PII leakage matched by a regex, output that contradicts a hard business rule?

This is also where you catch the silent failure mode, the agent that confidently returns a polished, well-formed answer that happens to be useless. A length and structure check won't catch "confidently wrong," but a did-it-actually-call-the-required-tool check often will. Cheap heuristics catch the dumb failures, and dumb failures are more common than anyone admits.

Layer 2: LLM-as-Judge on a Sample

Here you use a separate model to grade the agent's output against a rubric, faithfulness, helpfulness, did-it-do-what-the-user-meant, did-it-follow-policy. This is powerful and flexible, and it's how most teams scale qualitative judgment. It's also where people get sloppy.

An LLM judge is itself a probabilistic system that can be wrong, biased toward verbose answers, and inconsistent across runs. Treat your judge as a component that needs its own evaluation: periodically check its agreement with human raters, and if that agreement drops, fix the judge before you trust its verdicts. Anthropic's guidance on building evals and the broader literature on LLM-as-judge reliability both stress this, an unvalidated judge is a confident liar measuring a confident liar.

Because of cost and latency, Layer 2 runs on a sample, not everything. Which raises the obvious question of how you sample well.

Layer 3: Human Review on the Hard Cases

The smallest, most expensive layer. Humans review the cases the lower layers flagged as ambiguous, high-stakes, or low-confidence, plus a small random sample to catch what the automated layers are systematically missing. The output of human review isn't just a fix for that one case; it's a labeled example that should flow straight back into your golden dataset and your judge's calibration set. Human review that doesn't feed back into the system is just expensive babysitting.

Sampling Without Going Broke

The naive approach, evaluate a flat random 1%, wastes most of your eval budget on boring, obviously-fine runs while missing the rare disasters. Smart sampling is weighted toward signal:

• Uncertainty sampling. If your agent emits a confidence score, oversample the low-confidence runs. That's where the failures cluster.
• Novelty sampling. Oversample inputs that look unlike your training and eval distribution, the new, the weird, the out-of-distribution. These are early warnings of input drift.
• Stratified sampling by stakes. A refund agent should review a far higher fraction of $5,000 decisions than $5 decisions. Match your review rate to the cost of being wrong.
• Always keep a random baseline. A small uniform random sample is your control group. Without it, your weighted samples can convince you everything's fine in exactly the regions you forgot to weight.

The goal is statistical: you want enough volume in each meaningful segment to detect a real shift in quality before your customers do, without paying to grade traffic that tells you nothing new.

Closing the Loop: From Signal to Fix

A continuous-eval signal that nobody acts on is theater. The loop has to close, and closing it well is mostly an operational problem, not a technical one.

The mature pattern looks like this. Online evals run constantly and write scores to a time series. A drift-detection layer watches those series for statistically significant degradation, not "today was a bad day," but "task success has trended down 6 points over two weeks." When a threshold trips, it alerts the eval team (yes, that's a real role now inside serious GaaS companies). They triage: is this a model-version change, an input shift, a broken downstream tool, a prompt regression? Then the fix ships, and critically, the failing cases that triggered the alert get added to the offline suite, so the same failure can never silently recur. This is also why canary deployments and shadow mode pair so naturally with continuous evals: you run a new agent version against live traffic, score it online, and compare before you let it touch real customers.

That last step, every production failure becoming a permanent offline test, is what separates teams that get steadily more reliable from teams that play whack-a-mole forever. Your offline suite should be growing every week, fed by what production teaches you. A pre-launch eval is a snapshot. A continuous-eval program is a flywheel.

What This Means for GaaS Pricing and Trust

Here's the part most engineering-focused write-ups skip: continuous evaluation is a business model issue, not just an SRE one.

If you sell agents on per-outcome pricing, you charge when the task succeeds, then your definition of success is your billing engine, and continuous evaluation is literally how you meter revenue. Get the eval wrong and you either overcharge customers for failures or eat the cost of work you can't bill. The reliability number a GaaS vendor puts on its homepage isn't marketing fluff; it's a claim that has to be backed by a live measurement system, and increasingly enterprise buyers ask to see exactly how that number is computed before they sign. McKinsey's research on scaling generative AI in the enterprise keeps landing on the same point: the blocker to deployment is rarely raw capability and almost always trust, governance, and demonstrable reliability over time.

Continuous evaluation is how you manufacture that trust at scale. It's the difference between telling a buyer "it worked great in our demos" and showing them a live dashboard of how it's performing on traffic that looks like theirs, today. In a market where capability is converging and reliability is the real moat, the eval program isn't overhead. It's the product.

Insights Most People Overlook

Your eval suite is a depreciating asset, and nobody budgets for the maintenance. Teams treat the golden dataset as capital expenditure, build it once, done. In reality it decays like a physical asset because the input distribution it represents drifts away from reality. If you're not allocating ongoing engineering time to refresh the eval set from production traffic, your "passing" score is measuring how well your agent handles last quarter's world. Budget for eval maintenance the way you budget for dependency upgrades.

The judge needs a watchdog more than the agent does. Everyone obsesses over evaluating the agent. Almost nobody continuously evaluates the evaluator. An LLM-as-judge can drift, get silently upgraded by its provider, or develop a bias toward a particular answer style, and when it does, your entire quality signal is corrupted while every dashboard stays green. The most dangerous failure in a continuous-eval system isn't a bad agent; it's a broken judge confidently certifying a bad agent. Run judge-vs-human agreement as its own tracked metric.

Continuous evaluation can create a privacy liability that pre-launch testing never had. Pre-launch, you evaluate against synthetic or sanitized data. Online, you're scoring real customer data, often by piping it through a third-party judge model. That's a data-flow your security and compliance teams may not know exists. The eval pipeline is a back door into production data, and it deserves the same scrutiny as any other data processor. More than one team has built a beautiful online-eval system that quietly violated its own data-processing agreement.

Aggregate metrics hide the failures that actually churn customers. A blended 95% task-success rate feels great and tells you almost nothing. The 5% that fails is rarely random, it clusters in a segment, a use case, or an edge condition, and that cluster is often your highest-value or most vocal customers. Continuous evaluation only earns its keep if you segment the metric. The headline number is for the homepage; the segmented number is for survival.

The fastest decay often comes from a change you didn't make. Engineers instinctively look at their own deploys when quality drops. But in GaaS, the most common cause of sudden degradation is an upstream model update or a downstream tool change, things outside your repo entirely. Continuous evaluation is the only thing that catches this class of failure, because there's no commit in your history to point at. If your incident response always starts with "what did we ship," you'll waste hours on the failures where the answer is "nothing."

References

• Stanford HAI, Artificial Intelligence Index Report
• a16z, The State of Generative AI
• Anthropic, Research
• McKinsey QuantumBlack, Insights on Scaling Generative AI