The Eval Team: The New Role That GaaS Companies Are Quietly Building First

Table of Contents

• Why a Whole Team for "Just Testing"
• What the Eval Team Actually Does
• Eval Engineer vs. QA: Not the Same Job
• Where the Team Sits in the Org
• The Daily Work: A Realistic Week
• Who You Actually Hire
• The Tooling Stack the Team Owns
• How to Know the Team Is Working
• Insights Most People Overlook
• References

Why a Whole Team for "Just Testing"

For most of software history, "does it work?" was a deterministic question. You wrote a function, you wrote a test, the test passed or failed, and it kept passing until someone changed the code. Quality assurance was real work, but it was bounded work.

Agentic AI broke that contract. When you sell an agent as a service, priced per task, per resolved ticket, per booked meeting, you're not selling code that does the same thing every time. You're selling a probabilistic system that reasons over messy inputs, calls tools, and produces an outcome that might be subtly wrong in ways no assertion catches. The same prompt can produce a clean result on Monday and a confident, plausible, completely useless result on Thursday. That's not a bug you can reproduce by clicking the same button twice.

So the question shifts from "does it work?" to "how often does it work, on what kinds of inputs, and is that rate getting better or worse?" Answering that continuously, rigorously, with evidence a customer would accept, that is a full job. Several jobs, actually. And the GaaS companies that figured this out early stopped treating evaluation as a phase and started treating it as a team.

The economics force the issue. A traditional SaaS company can absorb a flaky feature; users file a ticket and move on. A GaaS company that bills per outcome is financially exposed to every failure. If your agent resolves 82% of support tickets and you priced as if it resolved 95%, you don't have a quality problem, you have a margin problem. Evaluation is where that gap gets measured before it shows up in churn.

What the Eval Team Actually Does

Strip away the titles and the eval team owns four things.

They define what "correct" means. This sounds trivial and is the hardest part. For a coding agent, is correct "tests pass" or "the code a senior engineer would have written"? For a sales-research agent, is a fabricated-but-plausible company detail a failure or a rounding error? The eval team writes these rubrics down, argues about edge cases, and turns fuzzy product intent into something measurable. Without this, every other metric is noise.

They build and maintain the eval suites. This is the dataset and harness work, golden datasets of real tasks with known-good outcomes, graders (sometimes deterministic, often LLM-as-judge, increasingly human-in-the-loop), and the infrastructure to run thousands of agent traces and score them. Anthropic's own guidance on building evals makes the point that the eval is often more valuable and more durable than the agent it measures; the agent gets rewritten, the eval persists.

They gate releases. When the underlying model changes, and it will, every few months, often without you choosing it, the eval team runs regression evals to catch silent degradation. The same applies to every prompt tweak, tool addition, or orchestration change. A mature eval team has the authority to block a deploy that drops task success rate, the same way a build break blocks a merge.

They publish the reliability number. Increasingly, GaaS companies put a real metric on the homepage, "94.2% task success across X production tasks." The eval team is the source of truth for that number, which means they're also the people who have to defend it to an enterprise buyer's procurement team. That accountability changes how they work.

Eval Engineer vs. QA: Not the Same Job

It's tempting to say the eval team is "QA for AI," and that framing costs companies real time, because it leads them to hire the wrong people and set the wrong expectations.

Traditional QA verifies known behavior against a spec. The pass/fail boundary is crisp, the goal is regression prevention, and a good QA engineer is rewarded for finding the one path that breaks. Eval work is statistical, not binary. An eval engineer isn't asking "did this break," they're asking "what's the distribution of outcomes, and did the distribution shift." A single failing trace might be perfectly acceptable noise; a two-point drop in success rate across a thousand traces is a five-alarm fire even if every individual run "looks fine."

The other big difference is that the eval team builds graders, and graders are themselves software that can be wrong. An LLM-as-judge that scores your agent's output is a model with its own error rate, its own biases, its own drift. So the eval team has a meta-problem QA never had: they have to evaluate their evaluators. They run the judge against human-labeled data, measure agreement, and recalibrate. This is closer to the work of a psychometrician or an experimental scientist than a tester.

That's why the QA-to-eval career pivot, while real, isn't automatic. The best eval engineers tend to come from a mix of backgrounds, some from QA, yes, but also from data science, ML research, and ironically from the domain itself. The person who used to do the work the agent now does often makes the sharpest evaluator, because they know what "good" looks like in their bones.

Where the Team Sits in the Org

There's no settled answer yet, and where the team reports tells you a lot about how serious a company is.

When eval reports into engineering, it tends to get treated as a testing function and starved of authority, it can flag problems but not stop ships. When it reports into research, evals get sophisticated but drift toward academic benchmarks that don't reflect production reality. The arrangement that seems to work best is a standalone reliability or eval function that reports high, often to a VP of Engineering or directly to the CTO, with explicit release-gating authority and a direct line to both research and product.

The reason is incentive alignment. Research wants to ship the capable new model. Sales wants to promise the high number. Product wants the new feature. The eval team is the only group whose job is to say "the data doesn't support that yet," and for that to mean anything, they can't be subordinate to the people they're checking. McKinsey's work on scaling generative AI in the enterprise repeatedly lands on the same theme: the organizations getting real value are the ones that built governance and measurement as first-class functions, not afterthoughts. Eval is that function with a sharper edge.

The Daily Work: A Realistic Week

Strip the abstraction away and here's what the work looks like.

Monday, the model provider ships a point release. The eval team's regression suite runs automatically overnight; by morning there's a report showing the new model improved on three task categories and regressed on one, specifically, multi-step tasks that require holding a constraint across several tool calls. Someone digs into the failing traces, confirms it's real and not a grader artifact, and files it. The agent stays pinned to the old model for those task types while research investigates.

Midweek, a customer escalation comes in: the agent "did something wrong." The eval team pulls the exact trace, replays it, and discovers the agent did the task it was asked perfectly, it just wasn't the task the user meant. That distinction (did-it-correctly vs. did-the-right-thing) becomes a new eval category, with five new golden examples added to the dataset so the suite catches that failure mode going forward.

Thursday is spent on grader calibration. The LLM-judge that scores summarization quality has been disagreeing with human labelers more often lately. The team samples a few hundred recent judgments, re-labels them by hand, finds the judge has gotten lenient on a specific kind of omission, and tightens the rubric.

Friday, they update the public reliability number and write the weekly reliability report that goes to the whole company. None of this is glamorous. All of it is load-bearing.

Who You Actually Hire

The first eval hire is usually a generalist who can do all of it badly and one part well, typically the dataset and grader engineering. As the team grows, the roles specialize into roughly three shapes.

The eval engineer builds and maintains suites, graders, and harnesses. Strong software skills, comfortable with statistics, allergic to vanity metrics. The domain evaluator, sometimes a contractor pool, sometimes full-time, provides ground truth in a vertical: the paralegal who knows what a correct contract redline looks like, the support lead who knows a real resolution from a deflection. The reliability or eval lead owns the methodology, defends the numbers externally, and holds the release-gating authority.

What you don't want is to hand evals to whoever has spare cycles, rotating week to week. Evaluation quality compounds. A good golden dataset built carefully over six months is a genuine moat, it's the reliability moat that's far harder to copy than raw capability, because it encodes thousands of small judgments about what "good" means in your specific domain. A model competitor can match your agent's capability in a release cycle; they can't easily reconstruct two years of accumulated, domain-specific eval data.

The Tooling Stack the Team Owns

The eval team is usually the heaviest internal user of agent observability tooling, and often the team that decides which platform the company buys. The category is consolidating fast, tracing tools that capture every step of a multi-step run, eval platforms that manage datasets and run graders at scale, and monitoring layers that watch production for drift.

A few capabilities are non-negotiable. End-to-end tracing so any production run can be reconstructed and replayed exactly, the replay problem is brutal with nondeterministic agents, and a team that can't recreate a failure can't fix it. Dataset versioning, because your golden data is an asset and you need to know which eval ran against which version. Both offline and online eval, pre-launch suites and continuous production evaluation, because pre-launch numbers always overstate real-world reliability. And grader management, since LLM-judges need their own versioning and calibration history.

The build-vs-buy question is live and unsettled. Plenty of teams start with homegrown harnesses and graduate to a platform once trace volume and dataset complexity overwhelm a spreadsheet-and-scripts setup. Either way, the eval team owns the decision, because they live in these tools every day.

How to Know the Team Is Working

The honest signal isn't a dashboard full of green. A well-functioning eval team makes the company less confident in the right ways, it surfaces the failure modes everyone was politely ignoring, replaces optimistic guesses with measured rates, and occasionally kills a launch that sales already promised.

Concretely, you're looking for a few things. The reliability number is reproducible and survives external scrutiny. Model upgrades stop being scary because regression is caught automatically. Customer escalations turn into new eval cases rather than recurring fires. And the gap between your benchmarked reliability and your real-world reliability shrinks over time, that closing gap is the clearest evidence the evals reflect reality.

If your eval team is just rubber-stamping ships and the number only ever goes up, that's not success. That's a team that isn't being allowed to do its job.

Insights Most People Overlook

The eval team's real product is institutional doubt, and that makes them politically fragile. Every other function is incentivized toward optimism, ship it, sell it, demo it. The eval team's value comes precisely from being the voice that says "not yet." Founders who don't actively protect that role watch it get quietly defanged: the number stays, but the authority to block a ship erodes until evals become decoration. The teams that stay effective are the ones where leadership treats a blocked launch as the system working, not the team being difficult.

Your evals are a more durable asset than your agent. The orchestration logic, the prompts, even the model will all be replaced, often within a year. The carefully built golden dataset and the rubrics encoding what "good" means in your domain outlast all of it. Smart GaaS companies account for this: they invest in eval infrastructure as a long-term asset and treat the agent itself as comparatively disposable. This inverts the usual instinct to pour everything into the agent and bolt on evals later.

LLM-as-judge quietly imports a second model's failure modes into your reliability number. Most teams celebrate when they automate grading with an LLM-judge, because human review doesn't scale. But the judge is a model with its own drift, biases, and blind spots, and when it degrades, your headline reliability number can move for reasons that have nothing to do with your agent. The teams that get burned are the ones who set up the judge once and trust it forever. Calibrating the evaluator against human ground truth has to be a recurring ritual, not a one-time setup.

The best evaluator is often the person whose job the agent replaces. Companies instinctively staff eval teams with engineers, but the sharpest ground-truth judgments come from domain practitioners, the support agent, the paralegal, the analyst. They catch the "did it correctly but did the wrong thing" failures that look fine to an engineer reading a trace. This creates an unexpected and somewhat uncomfortable internal career path: the humans being automated become the humans defining whether the automation is good enough.

"Per-outcome pricing" only works if evals are honest, which creates a structural tension between the eval team and the revenue model. When you bill per resolved ticket, the pressure to define "resolved" generously is enormous and comes straight from the P&L. The eval team is the only thing standing between an optimistic definition and a refund-generating one. That tension is healthy, but it means the eval function is, in a real sense, a check on the business model itself, which is exactly why it can't report to the people who own the revenue number.

References

• Building Effective Agents, Anthropic
• The State of AI, McKinsey & Company