The First Credible GaaS Benchmark Report: What It Actually Needs to Contain

Table of Contents

• Why GaaS Has No Benchmark Yet
• The Core Metrics a Credible Report Must Carry
• Cost: Report the Fully Loaded Number or Don't Bother
• Reliability: Success Rate Is a Lie Without Segmentation
• The Methodology Section That Makes or Breaks Credibility
• What a Sample Scorecard Row Looks Like
• Who Should Publish It, And Who Shouldn't
• Insights Most People Overlook
• Frequently Asked Questions
• Conclusion
• References

Why GaaS Has No Benchmark Yet

Buy a CRM and you can pull up win rates, switching costs, and a decade of Gartner peer reviews before lunch. Buy an agent that "handles your tier-1 support autonomously" and you get a sales engineer, a sandboxed demo on cherry-picked tickets, and a per-resolution price that quietly assumes a success rate nobody will commit to in writing.

The gap exists for a structural reason. SaaS benchmarks measure a product that behaves the same way for every customer, the software either has the feature or it doesn't. Agents are probabilistic. The same agent on the same task can succeed at 91% for one customer's data shape and 64% for another's, and the cost per attempt swings with model prices, retry behavior, and how many tools the workflow fans out to. You can't benchmark a coin flip the way you benchmark a spreadsheet.

The existing academic benchmarks, think the agent leaderboards built around things like SWE-bench or WebArena, measure capability in a vacuum. They answer "can a frontier model resolve this GitHub issue," not "what does it cost in dollars when this vendor's production agent resolves a real customer's issue, including the three times it failed first." That dollars-and-failures gap is precisely the territory a GaaS benchmark has to own, and it's why borrowing an academic leaderboard wholesale produces a number that looks rigorous and means nothing to a buyer. Andreessen Horowitz's writing on how AI agents will change the way software is priced frames the pricing shift well, but pricing models without measured outcomes are just hypotheses.

So the first credible report isn't a ranking. It's a measurement protocol. Get the protocol right and the rankings follow.

The Core Metrics a Credible Report Must Carry

Strip away the dashboard theater and there are four families of numbers that actually decide whether an agent is worth buying. A report missing any one of them is incomplete, and most of what passes for "benchmarking" today reports the easy two and skips the hard two.

Cost per completed task (CPCT). Not cost per attempt, not cost per token, cost per successful outcome, with all failed attempts amortized into the price. This is the unit metric the whole category is converging on, and a report that leads with anything else is measuring the wrong thing.

Reliability, segmented. Task completion rate and success rate are different numbers (a task can complete and still be wrong), and both are meaningless as single aggregate figures. They need to be sliced by task difficulty and input variance.

Human-intervention rate. What fraction of runs needed a person to step in? This is the quiet killer of agent ROI and, increasingly, the best leading indicator of churn, when humans keep rescuing the agent, the customer is already mentally drafting the cancellation email.

Latency and its dollar cost. Time-to-completion matters, but only paired with what that speed cost. An agent that resolves a ticket in nine seconds for $4.10 and one that takes ninety seconds for $0.30 are different products for different buyers; a benchmark that reports only the clock or only the bill hides the trade-off.

Everything else, autonomy grade, time-to-value, net revenue retention, is a derivative or a downstream consequence of these four. Nail the core, and the rest become reportable.

Cost: Report the Fully Loaded Number or Don't Bother

Here is where almost every early attempt will cheat, usually without meaning to.

A vendor benchmarks their support agent and reports "$0.22 per resolution." Sounds great. What that number quietly excludes: the 18% of tickets the agent attempted, failed, and silently retried (each retry burning a fresh stack of model calls); the vector-database lookups and the three tool calls per run; the "thinking" tokens the reasoning model spent before producing a single visible output; and the idle compute holding context warm between conversational turns. The honest number, the one a CFO would recognize on an invoice, is closer to $0.70. The reported figure is a lab result; the real figure is a production bill.

A credible report forces the fully-loaded calculation and shows the decomposition:

• Model inference, including reasoning/thinking tokens, which on modern reasoning models can dwarf the visible output and are easy to leave out by accident.
• The retry tax, every failed attempt before success, counted. This is the single most underreported line item; one nominal task can balloon into dozens of model calls when an agent gets stuck in a loop, and the difference between a 10% and a 30% retry rate can double your effective cost per outcome.
• Tool and API calls, search, code execution, third-party vendor models the agent orchestrates. An agent that calls three other vendors' models has a gross-margin structure totally unlike a single-model agent, and the report should make that visible.
• Memory and storage, vector stores, conversation history, the cost of remembering.
• Idle / warm-context cost, the compute a long-running agent burns while waiting, which vendors are especially good at hiding.

The presentation rule that separates a credible report from a marketing asset: report cost as a distribution, not an average. Per-task pricing makes the mean nearly useless because spend is lumpy and right-skewed, a handful of runaway runs can be 50x the median. Buyers need the p50, the p90, and the p99 so they can budget for the worst-case spend scenario, not the happy path. A single "average cost per task" is the benchmarking equivalent of telling someone the average depth of a river before they wade across.

Common mistake to call out explicitly: benchmarking on a frontier model's list price when the vendor runs reserved or open-weight inference in production (or vice versa). The compute substrate is the COGS. A report that doesn't pin down what hardware and pricing tier each cost figure assumes is comparing numbers that aren't comparable.

Reliability: Success Rate Is a Lie Without Segmentation

"94% success rate" is the headline number every vendor wants printed. It's also the number most likely to mislead, because it's an average over a task mix the vendor chose.

Here's the failure mode in practice. A coding agent posts 94% on a benchmark suite. Dig in and the suite is 80% trivial tasks (rename a variable, add a log line) where the agent hits 99%, and 20% genuinely hard tasks (cross-file refactors with failing tests) where it hits 74%. Your actual workload is mostly hard tasks. Your real-world success rate is going to land near 74%, not 94%, and you'll discover that after you've signed, when the human-intervention rate quietly eats the savings.

A credible report fixes this by refusing to publish a single aggregate. Instead:

Segment by difficulty tier. Easy / medium / hard, with the criteria for each tier defined and disclosed. Report success rate per tier. Now a buyer can map the agent's profile onto their own workload mix instead of trusting a blended number.

Separate completion from correctness. Did the agent finish and did it finish correctly are different events. An agent can confidently close a support ticket with a wrong answer, that's a completed task and a failed outcome. The report needs both rates, because the gap between them is exactly where silent quality erosion lives.

Report the human-intervention rate alongside success. A 90% success rate that required a human to catch and fix 25% of runs is not a 90% agent, it's a 65%-autonomous agent wearing a 90% costume. Pair the two numbers or you've described half the product. This intervention rate also doubles as the category's best early churn signal: rising human rescues predict cancellation long before the usage graph turns down.

Show variance across input shapes. Run the same task type across messy real-world inputs, not just the clean canonical examples. The spread between best-case and worst-case input is often where a vendor's "reliability" claim quietly falls apart.

The better approach, in one line: report reliability as a matrix (difficulty × correctness × intervention), never as a single percentage. A scalar success rate is a marketing number. A reliability matrix is a benchmark.

The Methodology Section That Makes or Breaks Credibility

A benchmark's authority lives entirely in its methodology disclosure. If readers can't reproduce it, it's an opinion with a chart. The well-worn standard here is the kind of structured documentation Google's research community pushed with its Datasheets for Datasets framework, provenance, composition, and known limitations stated up front. A GaaS report needs the equivalent for its task suite.

Non-negotiables for the methodology section:

• The task suite is public, or at least audited. If the tasks are secret, the vendor can teach to the test and nobody can check. At minimum a third party should attest that the suite wasn't leaked to vendors in advance, agent benchmarks are notoriously easy to contaminate when training data overlaps the test set.
• Pricing assumptions are pinned. Token prices move weekly. The report must state the exact model versions, the exact per-token prices, and the date they were captured, because falling token prices famously haven't lowered agent bills, and a benchmark that doesn't timestamp its cost basis ages into nonsense within a month.
• Run counts and confidence intervals. Agents are stochastic; a single run proves nothing. Every figure needs an n and a spread. "94%" from five runs and "94%" from five hundred runs are not the same claim.
• Conflict-of-interest disclosure. Who funded the benchmark, who built the task suite, whether any benchmarked vendor had input. This is the line most "industry reports" cross, and it's why they read as marketing.
• Versioning. Agents ship updates weekly. Every score must be stamped with the agent version and date. An unversioned agent benchmark is a photo of a moving car with no timestamp.

Skip these and you don't have a benchmark, you have a leaderboard nobody should trust, which is exactly the situation the category is in today.

What a Sample Scorecard Row Looks Like

Concrete beats abstract. Here's what one row of a credible GaaS benchmark, a single vendor's support agent, should actually contain, so the format is unambiguous:

Field	Value
Agent / version	VendorX Support Agent v3.2 (captured 2026-06-18)
Task suite	OpenSupportBench v1, 600 tickets, public
Difficulty mix	Easy 40% / Medium 40% / Hard 20%
Success rate (correct outcome)	Easy 97% · Medium 88% · Hard 71%
Completion vs. correctness gap	6 pts (some tickets closed wrong)
Human-intervention rate	14% of runs
CPCT (fully loaded)	p50 $0.31 · p90 $0.94 · p99 $3.80
Cost decomposition	Inference 58% · retries 21% · tools 14% · memory/idle 7%
Median latency	22s (p99: 140s)
Inference substrate	Reserved capacity, open-weight 70B + frontier fallback
Runs per figure	n = 600, 3 seeds

Notice what this row makes impossible to hide: the hard-task success rate, the retry tax as a named percentage of cost, the tail of the cost distribution, and the substrate assumption. A vendor can't wave away a 71% on hard tasks or a $3.80 p99 once they're printed in the same row as the flattering headline. That's the whole point.

Who Should Publish It, And Who Shouldn't

The first credible report will not come from a vendor benchmarking themselves; that's a press release. It also probably won't come from a legacy analyst firm applying SaaS frameworks to a category that breaks them, the temptation to reach for revenue-multiple thinking and seat-based comparisons is too strong, and those tools don't fit usage-based, probabilistic products.

The credible candidates are independent benchmarking consortia (the way MLPerf works for hardware, neutral body, public suite, vendor submissions audited against shared rules), well-resourced procurement teams pooling anonymized production data, or a research-led publication willing to spend real inference budget running the tests itself rather than accepting vendor-supplied numbers. The financial-rigor lens that publications like the Harvard Business Review's work on measuring AI's business value bring is useful here, the report should read like a unit-economics teardown, not a capabilities demo.

The tell for credibility is simple: did the publisher spend their own money running the agents, or did they accept the vendors' screenshots? Everything follows from that.

Insights Most People Overlook

1. The benchmark's biggest value is forcing cost honesty, not ranking vendors. Everyone assumes the point is a leaderboard. The deeper value is that a credible methodology forces every vendor to compute their own fully-loaded CPCT to respond, and many genuinely don't know their real per-outcome cost today because the retry tax and thinking-token spend live in dashboards nobody reconciles. The first report will quietly make vendors better at their own unit economics just by existing. The ranking is secondary; the disclosure discipline is the gift.

2. A "good" benchmark score can be a margin red flag, not a green one. A vendor posting suspiciously high reliability on hard tasks at a low price is often doing it by burning gross margin, running frontier models on every step, retrying aggressively, never caching. That's a great demo and an unsustainable business. The report should pair reliability with a cost-efficiency figure so buyers can spot vendors who are buying their benchmark scores with VC money. Today's 94%-at-$0.22 may be next year's price hike or shutdown.

3. Per-task pricing means the benchmark must report a distribution, or it's actively misleading. This is worth repeating as its own insight because it's the most common methodological sin. Reporting "average cost per task" on a right-skewed, lumpy distribution doesn't just lose information, it systematically understates what buyers will actually spend, because the long tail of runaway runs pulls real bills well above the median while the mean stays seductively low. A benchmark that reports only means is optimistic by construction. p90 and p99 aren't nice-to-haves; they're the honesty check.

4. Human-intervention rate belongs in the cost column, not just the reliability column. Most frameworks file intervention under "quality." But every intervention is a human salary spent rescuing the agent, it's a cost, and often the largest hidden one. A truly credible report would convert intervention rate into a dollar figure (intervention rate × average loaded human-minutes × labor cost) and fold it into the fully-loaded CPCT. Do that and a lot of "cheap" agents stop looking cheap. The agent that costs $0.30 per task but needs a human one run in four can be more expensive than the $0.90 agent that almost never does.

5. The first credible report will be obsolete in 90 days, and that's a feature, not a bug. Agents version weekly; token prices move; substrates shift from frontier to open-weight overnight. A static annual PDF is the wrong artifact entirely. The credible format is a living, versioned, continuously-rerun benchmark with timestamps on every number, closer to a status page than a report. Anyone planning a glossy yearly publication has already misunderstood the category's clock speed.

Frequently Asked Questions

Why can't we just use existing academic agent benchmarks like SWE-bench? Because they measure capability, not delivered economics. They tell you whether a frontier model can solve a task in ideal conditions; they say nothing about a specific vendor's production cost per successful outcome, their retry behavior, their human-intervention rate, or what they charge. They're a necessary input to a GaaS benchmark, not a substitute for one.

Isn't "cost per completed task" just cost per token with extra steps? No, and conflating them is the central error. Cost per token ignores failed attempts, retries, tool calls, and the success rate itself. Two agents with identical token prices can have wildly different CPCT if one succeeds 90% of the time and the other 60%, the failures get amortized into every success. CPCT is the unit that actually shows up on the invoice.

How do you benchmark something probabilistic without it being noise? Run counts and confidence intervals. A single run is anecdote; a few hundred runs across multiple seeds and input shapes produces a distribution you can actually reason about. The benchmark reports the spread, not just a point estimate, and any figure without an n attached should be treated as marketing.

Won't vendors just teach to the test? They will if the suite is public and static, which is why methodology matters more than the scores. Mitigations: rotate a held-out portion of the suite, audit for training-data contamination, require run reproducibility, and disclose conflicts. A benchmark that can't defend against teaching-to-the-test isn't credible no matter how clean its charts look.

Should the report rank vendors at all, or just measure them? Measure first, rank cautiously. A single overall ranking re-introduces the aggregation problem the whole report exists to solve, the "best" agent depends entirely on your task-difficulty mix and your cost tolerance. A credible report gives buyers the matrix and lets them rank for their own workload, rather than pretending one number fits everyone.

Where does latency fit, is it really a top-tier metric? Yes, but only as a pair with cost. Latency alone favors agents that throw expensive parallel model calls at every step; cost alone favors slow, cheap agents. Reporting time-to-completion next to the dollar cost of that speed exposes the trade-off so buyers can choose the point on the curve that fits their use case.

Conclusion

The category doesn't lack benchmarks because the work is impossible. It lacks them because the honest version is uncomfortable, it forces fully-loaded costs, segmented reliability, and intervention rates into the same table as the headline numbers vendors prefer to show alone.

The most common mistake the first attempts will make is importing SaaS habits: single averages, capability scores divorced from dollars, static annual PDFs, and secret task suites. The better approach is a measurement protocol, not a leaderboard, cost as a distribution with the retry tax and idle compute included, reliability as a difficulty × correctness × intervention matrix, and a methodology section rigorous enough that a skeptic could reproduce it.

If you're a buyer, your practical next step is to demand these numbers from any agent vendor today, before a public benchmark exists: ask for fully-loaded p90 cost per completed task, hard-task success rate, and human-intervention rate, in writing. If you're a vendor, compute your own real CPCT now, including thinking tokens and retries, because when the first credible report lands, the gap between your demo number and your invoice number is going to be very public. The first organization to publish this protocol well won't just rank the market; it'll define how the category gets bought.

References

• Andreessen Horowitz, perspectives on AI agents and software pricing
• Google Research, Datasheets for Datasets and benchmark methodology work
• Harvard Business Review, measuring the business value of AI