When the Agent Did Something Wrong: The Forensic Challenge of Investigating an Agent's Decision

Table of Contents

• Why Agent Forensics Is Not Like Software Forensics
• The Anatomy of a Single Agent Decision
• The Five Things That Make Reconstruction Hard
• What You Actually Have to Capture
• The Replay Problem: Can You Re-run the Decision?
• Who Investigates, and With What Tools
• Designing for the Post-Mortem Before the Incident
• Insights Most People Overlook
• References

Why Agent Forensics Is Not Like Software Forensics

When a conventional program fails, the investigation has a comforting property: it's deterministic. Feed the same input to the same code, and you get the same output. Stack traces point at exact lines. Logs say what happened in order. A skilled engineer can usually reproduce the bug on a laptop and bisect it down to a commit.

Agentic systems break almost every one of those assumptions. The "program" is a large language model whose output for the same prompt can differ run to run. The control flow isn't written in your codebase -- it's generated at inference time as the model decides which tool to call next. And the inputs aren't a tidy function signature; they're a sprawling context window stuffed with system prompts, retrieved documents, prior conversation, tool outputs, and whatever the user typed. Change any token in that pile and the decision can flip.

This is why "investigating an agent's decision" deserves its own playbook within the broader conversation about trust, safety, and governance in Agentic AI-as-a-Service. It sits next to questions of liability and audit logging, but it's mechanically different. You're not just asking what the agent did -- you're trying to reconstruct a chain of probabilistic reasoning that may never occur the same way twice. Anyone who has read Anthropic's research on the limits of interpretability understands the uncomfortable core of it: even the people who build these models can't fully explain, token by token, why a given output emerged.

The Anatomy of a Single Agent Decision

To see why forensics is hard, you have to look at what "one decision" actually contains. Say a customer-support agent issues a $4,000 refund it shouldn't have. That single bad outcome decomposes into a surprising number of steps:

• The system prompt and policy text that were loaded that day (which may have been edited since).
• The retrieval step that pulled in the customer's order history -- including which documents the vector search returned, and which it didn't.
• The model's first reasoning pass, where it decided the refund was warranted.
• A tool call to look up the refund policy, and the tool's response.
• A second reasoning pass that interpreted that response.
• The final tool call to the payments API that actually moved money.

Six steps, and that's a simple case. A real production agent might chain twenty or more tool calls, sometimes spawning sub-agents, sometimes looping until a condition is met. Each model invocation has its own temperature, its own token sampling, its own context. The decision you're investigating isn't a point -- it's a trajectory. And the failure could live anywhere along it: a stale policy document, a retrieval miss, a hallucinated justification, a tool that returned ambiguous data, or a prompt injection buried three messages back. The chain-of-custody problem in multi-agent workflows makes this worse, because the agent that took the harmful action may not be the one that made the flawed judgment.

The Five Things That Make Reconstruction Hard

Non-determinism

The headline problem. Because LLM sampling is stochastic, re-running the exact same inputs may not reproduce the bad decision. You can set temperature to zero, but that doesn't fully eliminate variance -- floating-point non-associativity across GPU batches, model version updates, and provider-side changes all introduce drift. Investigators trained on deterministic systems instinctively try to "reproduce the bug" and discover the bug refuses to show up.

Context opacity

The decision depended on the exact context window. If you didn't log the fully-rendered prompt -- not the template, the rendered result with every retrieved chunk and every prior turn substituted in -- you can't reconstruct what the model actually saw. Most teams log the template and the user message and assume they can rebuild the rest. They usually can't, because retrieval results and tool outputs are ephemeral.

Model version drift

The provider may have silently updated the underlying model between the incident and your investigation. A decision made by model-v2.3 on Tuesday cannot be faithfully replayed on model-v2.4 on Friday. If your logs don't pin the exact model snapshot, your replay is investigating a different system than the one that failed.

The reasoning-vs-rationalization gap

When you ask a model to "explain why you did that," it produces a plausible-sounding justification -- but that justification is generated after the fact and may bear little relationship to the actual computation. This is one of the most dangerous traps in agent forensics. A post-hoc explanation feels like evidence. It isn't. It's another model output, equally capable of being wrong.

Distributed state

Agents touch external systems -- databases, APIs, message queues, other agents. The "decision" is entangled with the state of all those systems at that moment. A refund that was correct given the account balance at 2:14 PM looks insane given the balance at 4:00 PM. Without synchronized, timestamped snapshots across every dependency, you're reconstructing a crime scene where half the furniture has been rearranged.

What You Actually Have to Capture

The depressing truth is that most forensic failures are really logging failures -- the evidence was never recorded. Borrowing from the discipline of distributed-systems observability, Google's Site Reliability Engineering practices emphasize capturing enough signal to reconstruct events after the fact, and agent systems need an even richer trail. At minimum, a defensible agent decision record should capture:

• The fully-rendered prompt for every model call, not the template. Every token the model saw.
• The exact model identifier and version snapshot, including provider, model name, and the pinned version string.
• Sampling parameters -- temperature, top-p, seed if available, max tokens.
• The raw model output, including any chain-of-thought or tool-call structure, before your code parsed or post-processed it.
• Every tool call and its raw response, with timestamps, so you can see what data the agent acted on.
• Retrieval provenance -- which documents were returned, their scores, their source versions.
• The decision trajectory -- the ordered graph of steps, including sub-agent invocations and any loops.
• External system state at decision time, or at least references that let you reconstruct it.

This is heavier than typical application logging, and it raises real tension with data-retention and privacy obligations, because these logs contain everything the agent saw -- often including sensitive customer data. That's a governance problem to solve deliberately, not a reason to log less. Audit logs regulators will demand from GaaS vendors increasingly assume this level of granularity exists.

One practical note from the field: log the rendered prompt as a content-addressed artifact (hash it, store it once, reference the hash). Prompts repeat constantly, and storing the full rendered context for every call naively will bankrupt your logging budget within weeks.

The Replay Problem: Can You Re-run the Decision?

The dream of agent forensics is deterministic replay: feed the captured inputs back in and watch the decision happen again, step by step, like rewinding a recording. In practice you get three tiers of fidelity.

Tier 1 -- Trace inspection. You read the captured trajectory without re-running anything. This always works if you logged properly, and it's where 80% of investigations actually conclude. You can usually see the flawed step by reading the trace, the same way a pilot reads a flight recorder.

Tier 2 -- Approximate replay. You re-run the captured inputs against the same model version and observe whether the bad decision tends to recur. Because of non-determinism, you run it many times and look at the distribution of outcomes. If the agent issues the bad refund 70% of the time, you've found a systemic flaw, not a fluke. This statistical framing is the right mental model -- you're not asking "did it happen," you're asking "how often does it happen, and under what conditions."

Tier 3 -- Counterfactual replay. You change one variable -- swap a retrieved document, edit a policy line, remove a suspicious message -- and re-run to see if the decision flips. This is how you isolate root cause. It's the agentic equivalent of git bisect, and it only works if your capture was complete enough to reconstruct alternate contexts.

The teams that can do Tier 3 are rare, and they got there by treating replayability as a design requirement, not an afterthought. It connects directly to the explainability demand from enterprise buyers: "show me why" increasingly means "show me you can re-derive the decision under controlled conditions."

Who Investigates, and With What Tools

Right now this work falls into an awkward gap. It's too model-specific for traditional incident-response teams, too operational for ML researchers, and too high-stakes to leave to whoever happened to ship the agent. Mature GaaS organizations are starting to staff a hybrid role -- part SRE, part ML engineer, part forensic analyst -- which overlaps heavily with the emerging trust-and-safety function inside agent companies.

The tooling is immature but moving fast. LLM observability platforms (LangSmith, Langfuse, Arize Phoenix, Braintrust, and the OpenTelemetry GenAI semantic conventions) now capture traces structured around model calls and tool invocations. The good ones let you click into any span, see the rendered prompt, and fork a replay from that point. That last capability -- fork-and-replay from an arbitrary point in the trajectory -- is the single most valuable forensic feature, and it's still rare. If you're a GaaS buyer evaluating vendors, this is a sharp question to put in your security questionnaire: "Can you reconstruct and re-run any individual decision your agent made for my account, six months after the fact?" The answer separates serious operators from demoware. Industry analysts like Gartner have flagged AI trust, risk, and security management as a top priority precisely because this kind of post-hoc accountability is so often missing.

Designing for the Post-Mortem Before the Incident

You cannot retrofit forensic capability after a bad decision. The evidence is gone. So the entire discipline collapses into one principle: instrument for the investigation you hope you never run.

Concretely, that means a few non-negotiables. Pin and log exact model versions so a replay investigates the right system. Capture rendered prompts, not templates. Make every decision trajectory a first-class, queryable object -- not a smear across a dozen log files. Snapshot or reference external state at decision time. And critically, never trust a model's own explanation of itself as evidence -- treat it as a hypothesis to test against the trace, never as the conclusion.

There's an organizational dimension too. The most forensically prepared teams run pre-incident drills: they pick a past agent decision at random and ask an investigator to reconstruct it cold. If they can't, that's a logging gap discovered on a quiet Tuesday instead of during a regulatory inquiry. It's the same logic that drives chaos engineering, applied to accountability instead of availability. This kind of rehearsal pairs naturally with the "agent acted without authorization" incident playbook -- you want the muscle memory before you need it.

The economics here are unintuitive but real. Comprehensive decision logging looks expensive until you price a single un-investigable incident: the regulatory exposure, the customer trust hit, the inability to prove the agent didn't do the thing you're accused of. Forensic readiness isn't overhead. In a per-outcome GaaS pricing world, where the vendor is on the hook for results, it's table stakes for being insurable at all.

Insights Most People Overlook

The agent's explanation is the least reliable evidence in the room. Teams instinctively reach for "ask the model why it did that" because it's easy and the answer sounds authoritative. But a post-hoc rationalization is a fresh generation, untethered from the actual computation. The most common rookie mistake in agent forensics is mistaking a confident explanation for a root cause. Trust the trace, never the testimony.

Deterministic replay is mostly a myth, and chasing it wastes your investigation. Investigators trained on traditional software burn days trying to make the bad decision reproduce exactly. It won't, and that's not a bug in your tooling -- it's the nature of the system. The professional move is to abandon "reproduce it once" for "characterize the distribution": run it fifty times and report how often, and under what conditions, the failure surfaces. Probability, not certainty, is the deliverable.

Your biggest forensic risk is a silent provider model update. Most teams don't even know the model version changed underneath them, because the API endpoint name stayed the same. A decision becomes permanently un-replayable the moment the provider rotates the underlying snapshot. Pinning versions isn't a nice-to-have; it's the difference between an investigable system and an opaque one. Build alerting on model-version drift the way you'd build alerting on a dependency you can't roll back.

The retrieval layer is where most "the agent went rogue" stories actually begin. When a decision looks inexplicable, the cause is frequently not the model's reasoning but what got fed into it -- a bad chunk surfaced by vector search, a stale document, a poisoned record. Forensics that focuses only on the model's output misses the upstream contamination. Half the time the agent reasoned perfectly over garbage inputs.

Logging everything the agent saw recreates the breach you were trying to prevent. The complete forensic record is, by construction, a high-resolution copy of every piece of sensitive data the agent touched. That log is now one of the most attractive targets in your infrastructure. Forensic readiness and data minimization are in direct tension, and the teams that ignore that tension end up building the very honeypot they'll later have to explain to regulators.

References

• Anthropic -- Mapping the Mind of a Large Language Model
• Google SRE Book -- Monitoring Distributed Systems
• Gartner -- AI Trust, Risk and Security Management (AI TRiSM)