Eval Benchmarks for Long-Horizon Agentic Tasks: What Actually Measures Whether Your Agent Holds Up

Table of Contents

• Why Long-Horizon Is a Different Evaluation Problem
• The Benchmarks Worth Knowing
• SWE-bench and Its Verified Cousin
• GAIA: Real-World Multi-Step Reasoning
• WebArena and the Browser Agents
• TAU-bench: Conversation Plus Tools Plus Rules
• What the Headline Numbers Hide
• How to Build Long-Horizon Evals That Predict Production
• Metrics Beyond Pass Rate
• Insights Most People Overlook
• References

Why Long-Horizon Is a Different Evaluation Problem

There's a quiet assumption baked into most AI evaluation: that if a model gets the right answer on a hard question, it can be trusted to do hard work. For chatbots, that assumption mostly holds. For agents that run for thirty minutes, filing a multi-leg expense report, refactoring a codebase, reconciling an invoice against three systems, it falls apart.

The reason is compounding. If an agent has a 95% chance of getting each step right and a task requires twenty independent steps, its odds of completing the whole thing cleanly are roughly 0.95^20, or about 36%. That math is brutal, and it's the central fact every team selling agents as a service has to reckon with. A model that looks near-perfect on a single-turn benchmark can be a coin flip, or worse, on anything that resembles real autonomous work.

Long-horizon tasks introduce failure modes that simply don't exist in short ones. Agents lose the thread of the original goal and start optimizing for a sub-goal they invented halfway through. They get stuck in loops, calling the same tool with slightly different arguments forever. They hit a dead end and, instead of backtracking, confidently fabricate a result, the "silent failure" pattern that's become one of the most-discussed reliability problems in the field. None of these show up when you grade a one-shot answer. You only catch them when you evaluate the entire trajectory.

This is why the GaaS reliability conversation has shifted from "how smart is the model" to "how does this thing behave over time." Capability and reliability are different axes, and long-horizon evals are how you measure the second one.

The Benchmarks Worth Knowing

A handful of benchmarks have become the de facto reference points for agentic capability. Each one stresses a different slice of the long-horizon problem, and knowing what each actually tests, and doesn't, is the difference between an informed buyer and a leaderboard tourist.

SWE-bench and Its Verified Cousin

SWE-bench is the closest thing the industry has to a gold-standard long-horizon benchmark. It hands an agent a real GitHub issue from a real open-source Python project and asks it to produce a patch that makes the project's hidden test suite pass. This is genuinely hard: the agent has to navigate a large codebase, understand intent from a terse issue description, edit multiple files, and produce something that compiles and passes tests it can't see.

What makes SWE-bench valuable for long-horizon evaluation is that success is verified by execution, not by an LLM judge or string match. The patch either makes the tests green or it doesn't. That objectivity is rare and precious. The catch: the original SWE-bench had quality problems, some issues were underspecified or had broken tests, which is why SWE-bench Verified, a human-filtered subset, became the number people actually cite. When a vendor quotes a SWE-bench score, ask which variant, and ask whether they used the agentless or full-agent harness, because the scaffolding around the model can swing results by double digits.

GAIA: Real-World Multi-Step Reasoning

GAIA, introduced by researchers at Meta, Hugging Face, and others, takes a different angle. Its questions are conceptually simple for a human but require chaining several tools, web search, file reading, calculation, image analysis, to answer. A typical GAIA question might ask you to find a specific number buried in a PDF linked from a Wikipedia page, then do arithmetic on it.

GAIA is interesting because it deliberately resists the brute-force approach. You can't memorize your way through it, and you can't one-shot it with raw model knowledge. The agent has to plan, execute, and verify across multiple modalities. The downside for evaluation purposes is that GAIA answers are exact-match, which means an agent can do all the reasoning correctly and still fail on formatting, a real problem when you're trying to separate reasoning failures from presentation failures.

WebArena and the Browser Agents

WebArena puts an agent inside fully functional, self-hosted clones of real websites, a shopping site, a forum, a GitLab instance, and gives it tasks like "post a comment on the most recent issue in this repo" or "find the cheapest product matching these criteria and add it to the cart." Because the environments are real web apps with real state, the agent has to deal with multi-page navigation, forms, and the messy reality of DOM interaction.

WebArena exposes something the cleaner benchmarks hide: how badly agents degrade when the environment is unforgiving. Leading agents have historically scored well under 50% here, and the gap between that and human performance (which clears 78%) is a useful gut-check against the hype. If you're selling a web-navigating agent, your customers' workflows look a lot more like WebArena than like a tidy Q&A dataset.

TAU-bench: Conversation Plus Tools Plus Rules

TAU-bench, from Sierra, is the one I'd push GaaS teams to study hardest, because it most resembles a deployed vertical agent. It simulates a customer-service scenario where the agent must talk to a (simulated) user, call domain APIs, and, crucially, follow a set of business policy rules. A task might involve processing a return that's only allowed under certain conditions; the agent has to gather information, check the rules, and act correctly.

TAU-bench introduced a metric the field needed: pass^k, which measures whether an agent can succeed on the same task k times in a row. This is the reliability question stated plainly. An agent that passes a task once but only succeeds three times out of eight attempts is not deployable, and pass^k makes that visible in a way that a single pass@1 number never will. For anyone building toward a published reliability number on their homepage, this is the spirit of the right measurement.

What the Headline Numbers Hide

Here's the uncomfortable part. The benchmark scores vendors put on slides are, more often than not, the most flattering number that could be honestly extracted from the data, and sometimes not even honestly.

The first problem is contamination. Many of these benchmarks are built from public GitHub repos and public web content that almost certainly appeared in training data. When an agent "solves" a SWE-bench issue, you can't always be sure it reasoned its way there versus pattern-matched a fix it had effectively seen. Verified subsets and freshly collected test sets mitigate this, but the arms race between benchmark builders and training-set scrapers is permanent. As a16z and others have noted in their coverage of the state of AI agent infrastructure, evaluation is rapidly becoming the bottleneck that separates demos from products.

The second problem is harness leakage. A benchmark score is never just the model, it's the model plus the scaffolding: the prompt, the tool definitions, the retry logic, the planning loop. Two teams running the same model on SWE-bench can post wildly different numbers based purely on their harness. So a leaderboard rank often tells you more about who built the cleverest scaffold than about the underlying reliability you'd inherit.

Third, and most important for buyers: averaged pass rates wash out the distribution that actually matters. A 70% pass rate could mean "fails 30% of tasks a little" or "nails 70% perfectly and catastrophically destroys the other 30%." In a vertical where a false action is expensive, finance, healthcare, legal, the tail is the whole story. The cost of a false positive versus a false negative varies enormously by domain, and a single averaged number erases exactly the information you need to price that risk.

Public benchmarks, in short, are necessary but never sufficient. They're a screening filter, not a deployment decision.

How to Build Long-Horizon Evals That Predict Production

If public benchmarks can't tell you whether your agent is safe to ship, what can? A golden dataset of your own, built from real or realistic tasks in your actual vertical. This is the unglamorous work that separates GaaS companies that survive from the ones that demo well and then quietly churn.

Start by harvesting real failures. Every production incident, every escalation to a human, every customer complaint is a candidate test case. Freeze the initial state, the inputs, and the correct outcome, and add it to a suite that runs on every model change and every prompt change. This is the core of eval-driven development: you don't ship a change until it passes the suite, and the suite only ever grows.

For long-horizon tasks specifically, you need environments that hold state, not just static input-output pairs. A multi-step task can only be evaluated by letting the agent actually run through a sandboxed version of the workflow, a mock CRM, a replayable API, a synthetic database. The benchmarks above are worth studying precisely because they model how to build these stateful, executable environments. TAU-bench's simulated user and rule-checking is a template you can adapt to almost any vertical agent.

Then decide how you'll grade trajectories. Execution-based checks (did the test pass, did the database end in the right state) are the gold standard because they're objective, prefer them wherever the task has a verifiable outcome. Where outcomes are fuzzy, an LLM-as-judge can score whether the agent did what the user meant, but treat the judge itself as a component that needs its own validation against human labels. And critically, evaluate the path, not just the endpoint: an agent that reaches the right answer by taking a forbidden action (refunding a customer it wasn't authorized to refund) has failed, even if the final state looks correct. That "did the wrong thing correctly" failure mode is invisible to endpoint-only scoring.

Finally, run each task multiple times. The single biggest mistake in agent evaluation is treating a single run as signal. Because models are stochastic and long-horizon tasks compound that randomness, you need pass^k or pass@k style repetition to measure reliability rather than luck. If you can't reproduce a success, you don't have a feature, you have an anecdote.

Metrics Beyond Pass Rate

Pass rate is where teams start; it shouldn't be where they stop. A mature long-horizon eval reports a small panel of numbers that together describe behavior:

• pass^k (consistency): success on the same task k times running. This is the reliability metric, full stop.
• Step efficiency: how many tool calls or actions the agent took versus the minimum needed. Bloated trajectories cost money and signal an agent flailing toward a goal rather than planning toward it, directly relevant to the per-task economics of any GaaS offering.
• Recovery rate: when the agent hits an error, how often does it recover versus loop or give up? Long-horizon tasks are full of recoverable errors; an agent's recovery behavior often predicts production reliability better than its raw success rate.
• Time-to-failure distribution: for failed runs, where in the trajectory did it break? Early failures are usually planning problems; late failures are usually drift or context-window problems, and the fix is different for each.
• Cost per successful task: the metric that actually maps to a business model. An agent that succeeds 90% of the time but burns $4 in tokens per attempt may be less viable than one at 80% and forty cents.

The point of the panel is that no single number captures long-horizon reliability, and any vendor or internal team that reports only one is either naive or selling. Google DeepMind and other labs have increasingly framed evaluation as a multi-dimensional safety and capability problem rather than a single score, and that framing is exactly right for agents.

Insights Most People Overlook

The benchmark you should fear is the one your customer hasn't written yet. Every published benchmark measures a task someone thought to formalize. Real production failures cluster in the weird, undignified edge cases nobody benchmarks, the malformed input, the half-completed prior session, the user who changes their mind mid-task. Your golden dataset's value is proportional to how much of that mess you've captured, and public benchmarks capture almost none of it.

Higher capability can lower reliability on long-horizon tasks. This is counterintuitive and real. More capable models are often more willing to take initiative, which means they're more likely to take an unauthorized but plausible-looking action rather than stopping to ask. On rule-constrained tasks like TAU-bench, a "smarter" model can score worse precisely because it improvises where a more cautious one would escalate. Capability and obedience are not the same virtue.

Pass^k exposes that most "deployed" agents are running below their advertised reliability. A vendor quoting 85% pass@1 might be at 50% pass^4, meaning fewer than half of customers asking for the same thing four times get four good answers. The gap between pass@1 and pass^k is the single most honest number in agent evaluation, and almost nobody publishes it. When you're evaluating a GaaS vendor, asking for their pass^k is the fastest way to find out how seriously they take reliability.

Saturated benchmarks are worse than no benchmark. Once a benchmark hits 90%+ across the leading models, it stops discriminating, everyone looks the same, and the remaining 10% is often noise or contamination artifacts rather than real capability differences. SWE-bench's rapid climb means the interesting evaluation has already moved to harder, longer, less saturated tasks. If a vendor is still bragging about a saturated benchmark, they're telling you they stopped paying attention.

The eval harness is a product, not a script. Teams treat their evaluation setup as throwaway tooling and then wonder why their numbers don't reproduce. The harness, environments, graders, repetition logic, trajectory logging, is core infrastructure that deserves the same engineering rigor as the agent itself. The GaaS companies pulling ahead on reliability are the ones who staffed an eval team and built the harness like it mattered, because it does.

References

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• GAIA: A Benchmark for General AI Assistants (Hugging Face)
• WebArena: A Realistic Web Environment for Building Autonomous Agents
• TAU-bench: A Benchmark for Tool-Agent-User Interaction (Sierra Research)
• Andreessen Horowitz: AI Agent Infrastructure and the State of the Market
• Google DeepMind: Responsible AI and Evaluation Research