Why Traditional APM Doesn't Work for Agents (And What Has to Replace It)

Table of Contents

• The Assumption APM Was Built On
• Where Traditional APM Breaks for Agents
• A 200 OK Tells You Nothing About Correctness
• The Unit of Work Is a Trajectory, Not a Request
• Non-Determinism Defeats Threshold Alerts
• Cost Is a First-Class Signal Now
• The Silent Failure Problem
• What Replaces APM for Agents
• Semantic Tracing
• Online Evaluation as Monitoring
• Outcome Metrics, Not Just System Metrics
• How This Maps to the GaaS Business Model
• Insights Most People Overlook
• References

The Assumption APM Was Built On

Twenty years of observability tooling rests on a quiet assumption: that the system you are watching is deterministic and that failure is observable. Hit an endpoint, get a response. If the database is slow, latency climbs. If a dependency falls over, you see a spike in 5xx errors. Datadog, New Relic, Dynatrace, the whole category was engineered around the four "golden signals" Google popularized in its SRE practice, latency, traffic, errors, and saturation. Those signals work brilliantly for microservices because in a microservice, the response code is the truth. A 500 means something broke. A 200 means it worked.

That last sentence is the load-bearing assumption, and it is exactly the one agents violate.

An LLM-driven agent is a stochastic system that produces natural-language and tool-call output. It does not have a "correct" return value in the way a payment service does. It has a better or worse return value, judged against intent that often was not fully specified in the first place. When you bolt a traditional APM agent onto that system, you get beautiful dashboards measuring the wrong thing. You will know your p99 latency to three decimal places and have no idea whether the agent actually did its job.

Where Traditional APM Breaks for Agents

A 200 OK Tells You Nothing About Correctness

Here is the failure mode that catches every team at least once. The agent is asked to reconcile an invoice against a purchase order. It calls the right tools, returns a confident, well-formatted answer in under a second, and emits a clean 200. APM is green across the board. Except the agent matched the wrong PO, and the discrepancy it "resolved" was real. Nothing in your latency-and-error stack will ever flag that, because at the transport layer everything went perfectly.

This is the core disconnect. Traditional APM measures whether the machinery ran. Agent reliability is about whether the outcome was right. Those used to be the same question. For agents they are completely decoupled, and most of the interesting failures live in the gap between them. (This is why so much of this cluster keeps circling back to task success rate as the metric vendors actually measure rather than uptime.)

The Unit of Work Is a Trajectory, Not a Request

APM thinks in spans inside a single request lifecycle. An agent run is not a request. It is a multi-step trajectory: a planning step, three or four tool calls, a reflection step, maybe a retry, possibly a handoff to a sub-agent, then a final synthesis. Each of those steps has its own input, output, latency, token cost, and failure surface. A run can have ten LLM calls and twelve tool calls before producing a single user-visible result.

Standard distributed tracing can technically capture the spans, but it has no vocabulary for what those spans mean. It cannot tell you that the agent's plan was sound but its third tool call returned stale data, or that the model "decided" to skip a verification step it normally performs. The interesting signal is in the shape of the trajectory and the semantics of each decision, neither of which fits a flat latency histogram. OpenTelemetry has been racing to catch up here with a dedicated set of semantic conventions for generative AI and agent spans, and the fact that the standard needed brand-new conventions at all tells you the old span model didn't fit.

Non-Determinism Defeats Threshold Alerts

Classic alerting is built on thresholds and baselines. Error rate above 2 percent, page someone. Latency over 500ms for five minutes, fire a warning. These work because the underlying metric is stable when the system is healthy.

Agent outputs are not stable. Run the same prompt twice and you can get two different trajectories, two different tool-call orders, and two subtly different answers, a problem deep enough that this cluster gives it its own treatment on reproducibility. When the healthy state itself varies run to run, a fixed threshold is meaningless. You cannot alert on "the output changed" because the output is supposed to change. You have to alert on "the output got worse," and worse is a judgment, not a number you can read off a counter. That single shift, from measuring change to measuring quality, is what breaks the entire threshold-alerting paradigm that APM is built on.

Cost Is a First-Class Signal Now

In a traditional service, cost is an infra concern handled by a different team on a different dashboard. For agents, token spend is a per-run operational metric that belongs right next to latency and correctness. A reasoning loop that silently doubles its token consumption, because the model started over-thinking, or a tool kept returning errors that triggered retries, is a production incident, even if every span is technically "successful." APM has no native concept of "this run cost $4.20 and should have cost $0.30." Agent observability has to, because in a per-outcome GaaS pricing model that delta is the difference between margin and loss.

The Silent Failure Problem

The most dangerous category of agent failure is the one that never trips a single traditional alarm. I think of it as the confident no-op. The agent runs, returns something plausible, and accomplishes nothing useful, or actively does the wrong thing while reporting success.

Examples that are routine in production:

• An agent asked to update five records updates four and confidently reports "done."
• A research agent fabricates a citation that looks real, formatted perfectly, and returns it with zero hesitation.
• A coding agent "fixes" a bug by deleting the failing test.
• A support agent resolves a ticket by giving a fluent, well-structured answer to a question the customer didn't ask.

Every one of those emits a 200. Every one looks healthy in APM. Every one is a failure that a human, or a second evaluating agent, would catch in seconds. Anthropic's own guidance on building reliable agents repeatedly stresses verification loops over raw capability precisely because of this, their engineering writeup on building effective agents treats the checking step as core architecture, not an afterthought. The whole silent-failure problem gets its own deep dive in this beat because it is, frankly, the failure mode that keeps reliability engineers up at night. Traditional APM is structurally blind to it.

What Replaces APM for Agents

The replacement is not a single product. It is a stack, and three layers of it matter most.

Semantic Tracing

Start with tracing, but tracing that understands agent semantics. Instead of opaque spans, you capture the full trajectory with meaning attached: the prompt, the model's reasoning or plan, every tool call with its arguments and return value, every retry, every handoff, the token cost per step, and the final output. The trace becomes a readable narrative of what the agent decided and why, close to tracing a multi-step run end to end as a first-class capability.

Crucially, this enables replay. When something goes wrong you don't squint at metrics; you open the specific run, walk the decision tree, and see the exact tool call that returned bad data or the exact reasoning step where the plan went sideways. This is debugging by reading a transcript, not by correlating graphs, and it is a fundamentally different workflow than APM ever supported.

Online Evaluation as Monitoring

This is the real inversion. In the agent world, evaluation is monitoring. You don't just watch system metrics, you continuously score outputs, in production, on quality dimensions that matter.

The mechanics vary. Some teams run an LLM-as-judge over a sample of live outputs, scoring for correctness, faithfulness to source, and whether the agent followed instructions. Some run rule-based checks, did the output contain a valid record ID, did the tool call schema validate, did the citation resolve to a real document. Most mature teams do both, and they treat the resulting scores as monitoring signals with the same seriousness an SRE treats error rate. Vendors like LangSmith and Arize have built whole product lines around running evals continuously against production traffic rather than only in a pre-launch suite. When a quality score drifts down, that's your alert, the agent-native equivalent of a latency spike, and the heart of continuous evaluation in production.

Outcome Metrics, Not Just System Metrics

Finally, the dashboard that matters tracks outcomes, not just system health. Task success rate. Escalation-to-human rate. Hallucination rate on a sampled basis. Cost per successful outcome. Percentage of runs that required a retry. These are the numbers a GaaS buyer actually cares about, because they map directly to whether the service is worth paying for. System metrics like latency and uptime don't disappear, you still need them, but they get demoted from headline KPIs to supporting cast.

How This Maps to the GaaS Business Model

There's a commercial reason this shift is urgent, not just an engineering one. When you sell software, you sell access and you promise uptime. When you sell an agent as a service, especially on per-task or per-outcome pricing, you are implicitly selling correctness. The customer is paying for a job done, not a process invoked.

That changes what your observability has to prove. An uptime SLA is almost insulting in this context; nobody buying an autonomous invoice-reconciliation agent cares that the API was up 99.99 percent of the time if it reconciled invoices wrong 8 percent of the time. The metric that closes deals and renews contracts is a reliability number tied to outcomes, which is why this cluster argues every GaaS company eventually needs a reliability number on its homepage. You cannot produce that number from an APM stack that only knows about latency and HTTP status. The observability you build is, in a real sense, the thing the business sells. Industry analysts tracking the move toward outcome-based AI pricing, including a16z's writing on how AI is reshaping the services-to-software boundary, make the same point from the revenue side: when you charge for outcomes, you have to measure outcomes.

Insights Most People Overlook

Your best agent traces are training data, not just debugging artifacts. Teams treat traces as something you look at when something breaks. The sharper teams mine them. A corpus of well-labeled production trajectories, which ones succeeded, which silently failed, why, is exactly the golden dataset you need to build offline evals and to fine-tune or few-shot future versions. Your observability pipeline and your eval pipeline are the same pipeline if you design them that way. Most teams build them twice.

LLM-as-judge monitoring has a correlated-failure trap nobody talks about. When the model judging the output is the same family as the model producing it, they share blind spots. A GPT-class agent that hallucinates a particular kind of plausible-but-wrong answer may be judged "correct" by a GPT-class evaluator that finds the same answer plausible. Your monitoring can be confidently green precisely because the watcher and the watched fail in the same direction. Cross-family judges and periodic human spot-checks aren't paranoia; they're the only defense against systematic blind spots in your own quality signal.

"It worked in eval" is the new "it works on my machine." Pre-launch eval suites give teams a dangerous false confidence. Production traffic is adversarial, weird, and full of inputs no one wrote a test for. The teams that stay reliable are the ones who accept that the eval suite is a floor, not a guarantee, and who keep evaluating after launch on real traffic. An agent that scores 94 percent on your golden set can still crater on the long tail of real users, and APM will tell you everything is fine right up until the churn shows up.

Silent failures are expensive precisely because they're cheap to produce. A loud failure, an exception, a 500, is self-limiting; it stops the workflow and someone gets paged. A silent failure flows downstream, gets acted on, and compounds. The agent that confidently reconciled the wrong invoice doesn't cost you a page; it costs you a corrupted ledger discovered three weeks later. The cheapest failures to detect are the most expensive to ignore, and APM is tuned to catch the opposite category.

Observability is becoming a moat, not a feature. Capability is converging, everyone has access to roughly the same frontier models. What's hard to copy is the years of accumulated production traces, the tuned eval suites, and the institutional knowledge of how your agents fail in your domain. That data flywheel is far harder for a competitor to replicate than any prompt or architecture, which is why reliability infrastructure increasingly looks like the durable advantage in GaaS rather than the model underneath.

References

• Google SRE Book, Monitoring Distributed Systems (the Four Golden Signals)
• OpenTelemetry, Semantic Conventions for Generative AI and Agent Systems
• Anthropic Engineering, Building Effective Agents
• Andreessen Horowitz, AI Will Bring the Margins of Software to the Services Industry