Measuring Agent Performance Against a Human Baseline, Fairly

Table of Contents

• Why The Human Baseline Is The Wrong Number And The Right One
• Pick Your Human: Which Baseline Are You Even Measuring Against
• The Four Conditions For A Fair Comparison
• What Humans Do Silently That Agents Get Credit For Skipping
• Scoring: The Rubric Is The Experiment
• Cost, Latency, And The Per-Outcome Economics Of A Fair Baseline
• A Practical Protocol You Can Run This Quarter
• Insights Most People Overlook
• References

Why The Human Baseline Is The Wrong Number And The Right One

There's a reflex in agent evaluation to reach for the human baseline as if it settles things. "Our support agent resolves tickets at 91%, human agents hit 88%." Done. Ship it. Put it on the homepage.

The problem is that the human baseline is simultaneously the most intuitive comparison a buyer can grasp and the easiest one to rig. A buyer doesn't have a mental model for "0.91 on our internal eval set." They absolutely have a mental model for "as good as a competent person, but it never sleeps." That's why the human baseline keeps showing up in GaaS sales decks, it translates an abstract reliability claim into something a procurement committee can argue about.

And that's exactly why it gets abused. The human number in most comparisons is a strawman: an overworked, undertrained, distracted human handling a task cold, while the agent was tuned for months on that exact distribution. Or worse, the human baseline is invented, a round number someone remembered from a 2019 report, never measured on the actual task.

So the honest framing is this: the human baseline is the wrong number to obsess over and the right number to anchor against. Wrong, because "beats the average human" tells you almost nothing about tail behavior, which is where agents actually hurt you. Right, because no buyer deploys an agent into a vacuum, they deploy it to replace, augment, or backstop a person, and that person's performance is the real status quo the agent has to beat.

This tension sits at the center of Beat 3 of the GaaS cluster on reliability and evaluation. If you can't measure the human fairly, every downstream claim, your task success rate, your reliability SLA, your "trust dashboard", inherits the bias.

Pick Your Human: Which Baseline Are You Even Measuring Against

Before you measure anything, decide which human you mean. These are not interchangeable, and conflating them is the single most common way agent-vs-human comparisons mislead.

The median worker. The typical person doing this job today, on a normal day, with normal fatigue. This is the most honest baseline for an automation ROI argument, because it's who the agent actually displaces or assists. It's also the one vendors quietly avoid, because medians include bad days.

The expert. Your best performer, well-rested, given time. Useful as a ceiling, if the agent beats the expert, that's a genuinely strong claim. But experts are rare and expensive, so beating the median while trailing the expert is the realistic and still-valuable outcome for most vertical agents.

The consensus panel. Several humans independently do the task; you take majority vote or adjudicated agreement. This is the gold standard for tasks with a defensible right answer, because it strips out individual idiosyncrasy. It's also how you discover that humans only agree with each other 80% of the time, which immediately reframes what "matching a human" can even mean.

The team-with-tools. A human who has the CRM, the knowledge base, the escalation path, and a colleague to ask. This is the fairest comparison for an agent that also has tools, because it controls for the fact that a lot of "human performance" is really human-plus-system performance.

The choice isn't cosmetic. An agent can beat the median worker and lose to the consensus panel on the same task set, and both numbers are true. Pick the baseline that matches the deployment claim you're actually making, state it plainly, and never let a sales deck silently swap a median for an expert between the methodology slide and the headline number.

The Four Conditions For A Fair Comparison

A comparison is fair when you could swap the agent and the human and the test wouldn't notice. Four conditions get you there.

Same Task, Same Distribution

The agent and the human must face the same inputs, drawn from the same real-world distribution, not a curated subset. The fastest way to inflate an agent number is to evaluate it on the cases it was built for while the human baseline came from a messier sample. Freeze a held-out set, sample it from production traffic, and route identical items to both. If you're building this set deliberately, the discipline overlaps heavily with building golden datasets for vertical agent evals (#118).

Same Context And Tools

If the human can see the customer's order history, the agent should too, and vice versa. A surprising amount of "the agent is smarter" turns out to be "the agent had the full context window and the human had a cluttered ticket." Equalize the information and the tooling, or label the asymmetry explicitly.

Same Scoring Rubric, Same Graders

Both outputs get scored by the same rubric, ideally by graders who don't know which is which. Blind grading matters more than people admit: evaluators who know they're looking at "the AI" grade it more harshly on style and more leniently on substance, or the reverse, depending on their priors. Hide the provenance.

Same Definition Of "Done"

Humans and agents fail differently, so "success" has to be defined at the outcome level, not the effort level. A human who escalates a hard ticket to a senior rep has succeeded, the customer got helped. An agent that confidently closes the same ticket with a wrong answer has failed, even though it "completed" the task. If your rubric rewards completion over correctness, you'll systematically overrate the agent. This is the same failure mode explored in the "silent failure" problem (#113), and it's worth designing the rubric specifically to catch it.

What Humans Do Silently That Agents Get Credit For Skipping

This is the part most comparisons miss, and it's where I'd spend the most attention.

Humans constantly do invisible reliability work. A support rep reads a confusing ticket, senses something is off, and pings the customer for clarification instead of guessing. A paralegal notices a contract clause looks unusual and flags it to an attorney rather than processing it. An accountant sees a number that doesn't reconcile and stops. None of this shows up as a "task" in your logs, it shows up as the absence of a bad outcome.

When you measure an agent against a human, you have to decide how to credit this behavior. If you only score the cases where the human produced a final answer, you've quietly excluded every case where the human's best move was to not produce an answer, and those are exactly the cases where naive agents fail. The agent that always answers will look more "productive" than the human who knows when to stop.

The fair move is to score the escalation as a correct action. A human who routes the right 8% of cases to a specialist and nails the other 92% should be credited at, effectively, 100%, because the customer outcomes are right. An agent that wants to compete with that human needs a working "escalate to human" path, and you need to measure whether it escalates the right cases. This is why escalation design and the autonomy dial (#149) aren't separate from evaluation, they're load-bearing parts of the baseline comparison itself. An agent without a credible escalation mechanism isn't competing with a real human; it's competing with a human who's been forbidden from asking for help.

There's a flip side worth naming. Humans also do silent bad work, rushing, copy-pasting stale answers, skimming. A truly fair baseline captures that too, which is one reason measuring real humans (not an idealized one) on the same held-out set tends to produce a lower, more honest human number than people expect.

Scoring: The Rubric Is The Experiment

Two teams can run "the same" agent-vs-human study and reach opposite conclusions purely because of how they scored. The rubric isn't an implementation detail. It is the experiment.

Start by separating dimensions that get bundled together. Correctness (is the answer right?), completeness (did it cover everything it should?), safety (did it avoid a harmful or out-of-policy action?), and appropriateness (did it do what the user actually meant, per #136?) pull in different directions. An agent can be more correct and less safe than a human on the same task. Collapsing these into one accuracy figure hides the tradeoff that the buyer most needs to see.

Then decide your aggregation honestly. Averages flatter agents because agents are consistent in the middle of the distribution and catastrophic in the tail. A human support rep's worst day is mildly unhelpful; an agent's worst day can be a confidently fabricated refund policy sent to a thousand customers. If your comparison reports only the mean, the agent wins and the buyer gets blindsided by the tail. Report the distribution, p50, p95, and the rate of catastrophic outcomes, not just the average. Anthropic's guidance on building effective evals for LLM applications makes a similar point: the useful signal often lives in the failure cases, not the aggregate.

For tasks with genuine ambiguity, measure inter-rater agreement among your human graders first. If your own experts only agree 78% of the time, then an agent that "matches a human" can mathematically never exceed roughly that ceiling on agreement-based scoring, and you've learned that the task itself has an irreducible subjectivity floor. That number is one of the most useful things a fair evaluation can surface, and it almost never appears in vendor benchmarks. The broader caution that public benchmarks overstate real-world reliability (#117) starts right here, with rubric and agreement choices that quietly inflate headline numbers.

Cost, Latency, And The Per-Outcome Economics Of A Fair Baseline

GaaS is sold per task or per outcome, so a performance comparison that ignores economics is incomplete. The fair question is rarely "is the agent more accurate than a human?" It's "at what accuracy, cost, and latency does the agent change the unit economics, and does the human baseline still win once you price in the agent's failure tail?"

This is where naive comparisons quietly cheat in the agent's favor. The agent's cost looks like pennies per task. The human's cost looks like a fully loaded hourly wage. But the agent's true cost has to include the verification layer, the human review of escalations, and the expected cost of its failures, a false approval, a mis-sent message, a compliance miss. McKinsey's work on the economic potential of generative AI is blunt that the value shows up only after you net out the oversight and error-handling overhead, not before.

Latency cuts both ways and deserves explicit accounting. An agent that resolves a ticket in nine seconds versus a human's nine minutes is a real and measurable advantage, but only if the nine-second answer is right. If 12% of those instant answers are wrong and generate a follow-up contact, your true resolution time and true cost both balloon, and the latency-reliability tradeoff (#137) flips against you. A fair comparison computes effective cost and effective latency: the numbers after you account for rework, not the happy-path numbers.

The cleanest way to express all of this to a buyer is cost-per-correct-outcome, holding quality fixed. Set a target quality bar, say, "must match the human consensus panel", and then ask what each option costs to clear that bar. That single framing dissolves most of the apples-to-oranges arguments, because it forces accuracy and economics into the same equation instead of letting a vendor win on one while quietly losing the other.

A Practical Protocol You Can Run This Quarter

Here's a protocol that's survived contact with real deployments, stripped to its load-bearing steps.

1. Define the unit of work and the "done" condition at the outcome level, including what counts as a correct escalation. Write it down before you look at any data.
2. Sample a held-out set from production, a few hundred items minimum, drawn from real traffic, never the agent's training distribution.
3. Choose and name your human baseline (median, expert, consensus panel, or team-with-tools) to match the deployment claim you intend to make.
4. Run both blind. Identical inputs to the agent and to the humans; equalize context and tools or label the gap.
5. Score with one rubric, by graders blind to provenance, across separated dimensions (correctness, completeness, safety, appropriateness). Measure human-grader agreement so you know your ceiling.
6. Report the distribution, not the mean, p50, p95, catastrophic-failure rate, plus cost-per-correct-outcome and effective latency.
7. Re-run it on a schedule. The model underneath shifts, traffic drifts, and a comparison that was fair in Q1 silently rots by Q3. Continuous evaluation, not a one-time bake-off, is the only version of this that stays honest.

You'll notice this protocol produces a less flattering agent number than a sales deck wants. That's the point. The team that publishes the honest number, distribution and all, is the team a serious enterprise buyer trusts, and trust is the actual currency of GaaS.

Insights Most People Overlook

• "Matches a human" has a hard ceiling you can compute in advance. If your human graders agree with each other only 80% of the time, no agent can exceed roughly 80% on agreement-based scoring, the remaining 20% is irreducible task subjectivity, not agent error. Most teams never measure their own inter-rater agreement, so they chase an accuracy target that's mathematically impossible and blame the agent for the gap.

• The fair human baseline is almost always lower than the one in the deck, and that helps you. When you measure real, fatigued, distracted humans on the actual held-out set, the human number drops below the idealized figure everyone quotes. That's not a problem to hide, it's a more defensible bar for the agent to clear, and a buyer who catches you using the inflated number will distrust everything else you claim.

• Crediting escalation correctly can flip the entire result. Score a human's "I'll ask a specialist" as success and the human number jumps, because escalation is what humans are good at. An agent without a working escalation path isn't being compared to a human, it's being compared to a human with their hands tied. The escalation design is part of the baseline, not a feature you bolt on later.

• Averages are the enemy of agent honesty. Humans fail gracefully and agents fail catastrophically, so the mean systematically favors the agent while hiding the tail risk that actually scares enterprise buyers. A vendor reporting only averages is, intentionally or not, optimizing for the wrong moment of the distribution.

• The human baseline ages, and nobody re-measures it. Teams measure the human once, enshrine the number, then keep comparing new agent versions against a frozen baseline while the humans themselves got faster, the tooling improved, or the traffic mix shifted. A baseline you measured a year ago is fiction today. Re-measuring the human is unglamorous and almost never funded, which is exactly why it's an edge.

References

• Building Effective Agents, Anthropic Engineering
• The Economic Potential of Generative AI: The Next Productivity Frontier, McKinsey & Company