Did the Agent Do What the User *Meant*? How to Actually Measure It

Table of Contents

• Why "Task Success" Quietly Lies to You
• Intent Is a Stack, Not a String
• The Four Failure Modes of Intent Misalignment
• A Workable Definition of "Did What the User Meant"
• How to Actually Measure It
• Step 1: Recover the Intent
• Step 2: Build an Intent-Resolved Rubric
• Step 3: Score With a Judge You've Calibrated
• Step 4: Track the Intent Gap Over Time
• The Metrics That Matter
• What Breaks This in Production
• Insights Most People Overlook
• References

Why "Task Success" Quietly Lies to You

Here's a true story pattern that repeats across every GaaS company I've talked to. A user asks a scheduling agent to "move my 3pm with Dana to later this week." The agent finds the meeting, reschedules it to Friday at 4:45pm, sends the invite, confirms. Task complete. Logs green. Success rate ticks up.

The user wanted morning. They had a flight Friday afternoon. They never said so, because to them "later this week" obviously meant the slots they normally take. The agent did exactly what was asked and exactly the wrong thing.

This is the uncomfortable truth that sits underneath the whole Reliability, Evaluation & Observability beat: completion is observable, correctness against intent is not, at least not directly. Most eval suites measure the thing that's easy to log. They check whether the API call returned 200, whether the output parsed, whether the agent reached a terminal state. Those are necessary. They are nowhere near sufficient. An agent can confidently complete the wrong task with a perfect success rate, and you'd never see it in your dashboards until the refund requests show up.

The reason this matters commercially, not just academically, is that GaaS buyers are increasingly pricing on outcomes. If you're charging per resolved ticket or per completed workflow, "completed" and "resolved the thing the customer actually needed" had better be the same number, and they usually aren't. Closing the gap between them is the entire job.

Intent Is a Stack, Not a String

The first mistake teams make is treating intent as equivalent to the prompt text. It isn't. A user's intent is layered, and most of the layers are never spoken.

There's the literal instruction ("move my 3pm"). There's the immediate goal (free up that slot / accommodate Dana). There's the standing context (this user never takes afternoon meetings, has kids' pickup at 3:30, hates double-booking). And there's the success condition the user is silently holding in their head, the picture of "done right" they'll compare reality against. The agent that nails the literal instruction but violates the standing context fails the user even though it "succeeded."

Anthropic's own guidance on building reliable agents makes a related point: effective agents need explicit context about the environment and the user's situation, not just the task, because the model can't infer what it was never told. Their engineering write-up on building effective agents is worth reading specifically for how much weight it puts on giving the model the right context rather than more capability. Measurement is the mirror image of that: you can only score intent satisfaction if you've captured the same context the agent needed to satisfy it.

So before you can measure "did it do what they meant," you have to make the meaning legible. You can't grade against an intent you never reconstructed.

The Four Failure Modes of Intent Misalignment

It helps to name the ways agents miss intent, because each one needs a different test. In practice I see four.

Literal-but-wrong. The agent follows the instruction precisely and produces an outcome the user didn't want. The scheduling example. These are the most dangerous because every surface-level signal says success. This failure mode gets its own dedicated treatment in the cluster, see the discussion of agents that do "the wrong thing correctly."

Under-specified guess. The instruction is ambiguous, the agent picks an interpretation, and it picks wrong. "Summarize this for the board", did they mean one page or one paragraph, financial or strategic framing? A well-designed agent asks. A confident one guesses and you find out later.

Scope drift. The agent does what was asked plus things that weren't, or wanders off the task across a long horizon. It was told to draft a reply and it also archived the thread, or it "helpfully" changed three unrelated calendar events.

Silent no-op. The agent technically completes but produces nothing useful, returns a vacuous summary, files an empty ticket, says "I've handled that" without having handled anything. The confident-but-useless failure is its own well-known problem and deserves separate eval coverage.

If your eval only catches outright errors and crashes, it catches none of these four. They all look like success to a naive monitor.

A Workable Definition of "Did What the User Meant"

Let me commit to a definition you can operationalize, because vagueness here is fatal.

An agent did what the user meant if a knowledgeable human, given the user's full intent stack and the agent's actual behavior, would judge that the outcome satisfies the user's success condition, including the unstated parts a reasonable agent should have inferred or asked about.

Two clauses do the heavy lifting. "Full intent stack" forces you to evaluate against context, not just the prompt. And "should have inferred or asked about" sets the bar correctly: the agent isn't punished for being unable to read minds, but it is punished for confidently guessing when a reasonable operator would have clarified. That distinction, guess vs. ask, turns out to be one of the most predictive signals of trustworthy agents, and it's measurable.

How to Actually Measure It

Here's the pipeline. None of the steps are exotic; the discipline is in doing all four rather than stopping at step three like most teams.

Step 1: Recover the Intent

You can't grade against an intent you haven't written down. For each eval case, capture three things separately: the raw instruction, the available context the agent had (or should have had), and the ground-truth intent, a short statement of what the user actually wanted, written by someone who knows.

For synthetic evals you author the intent up front. For real production traffic, you recover it after the fact: from the user's follow-up behavior (did they accept the output, edit it, redo the task, complain?), from explicit thumbs/feedback, or from a human reviewer reading the conversation. Implicit signals are gold here, a user who immediately re-issues a corrected instruction is telling you intent was missed, no survey required.

Step 2: Build an Intent-Resolved Rubric

Generic rubrics ("is the answer helpful?") don't measure intent satisfaction; they measure vibes. Instead, decompose each intent into checkable criteria for that case:

• Must-haves: conditions that, if violated, mean failure regardless of everything else ("meeting moved to a morning slot," "refund issued to the original card").
• Should-haves: quality conditions that degrade but don't fail the outcome ("included a polite note to Dana").
• Must-nots: scope and safety boundaries ("did not touch any other calendar event," "did not expose the customer's full card number").

This turns a fuzzy judgment into a structured one. It also makes disagreement productive: when humans split on a case, it's almost always because a must-have was implicit and unwritten, so you write it down and your rubric gets sharper. Vendors who measure task success well, as documented in OpenAI's evals cookbook on grading model outputs, lean heavily on this kind of criterion-level grading rather than single holistic scores.

Step 3: Score With a Judge You've Calibrated

At volume, humans can't grade every run, so you use an LLM-as-judge that takes the intent statement, the rubric, and the agent's full trace, then scores each criterion. This works, but only if you've earned the right to trust it.

Calibrate the judge against a human-labeled gold set before you believe a single one of its numbers. Measure judge-vs-human agreement (Cohen's kappa, not raw accuracy, so you account for chance). If the judge can't match your reviewers on the gold set, fix the judge, better rubric, better prompt, a stronger model, before you scale it. Re-run that calibration whenever you change the judge model, because the judge is itself an agent subject to drift. And give the judge the trace, not just the final answer: half of intent failures (scope drift, silent no-ops) are only visible in the steps, not the output.

Step 4: Track the Intent Gap Over Time

Now compute the number that actually matters: the intent gap, the spread between your instruction-completion rate and your intent-satisfaction rate. A 96% completion rate paired with an 81% intent-satisfaction rate means roughly one in six "successful" runs did the wrong thing. That 15-point gap is your real reliability story, and it's the one to put in front of customers and on your own homepage reliability number.

Trend it per workflow, per intent category, and per model version. When you swap the underlying model, the gap is exactly where regressions hide, completion can hold steady while intent satisfaction quietly craters, which is why regression testing against model changes belongs in the same pipeline.

The Metrics That Matter

A few concrete metrics worth standing up, beyond plain task success:

• Intent-satisfaction rate. Share of runs the judge (calibrated) marks as satisfying all must-haves and no must-not violations. Your headline number.
• Intent gap. Completion rate minus intent-satisfaction rate. The single most honest reliability signal you have.
• Clarification-appropriateness rate. Of ambiguous cases, how often did the agent ask instead of guessing, and how often did it ask when it shouldn't have (annoying over-clarification counts against it too).
• Correction rate. Share of production runs where the user immediately edits or redoes the work. A cheap, real-world proxy for intent failure that needs no labeling.
• Scope-violation rate. How often the agent did something outside the intended scope. Catches drift and "helpful" overreach.

Notice none of these require you to know the "right" output in advance for every case. They measure satisfaction of a condition, which is a much more achievable bar than exact-match correctness.

What Breaks This in Production

A few honest cautions, because this is harder than a clean methodology makes it sound.

Intent is sometimes genuinely unknowable from the data you have, and you'll be tempted to label those cases as agent failures when they're really specification failures. Separate them. If a reasonable human reviewer also couldn't tell what the user meant, that's a product/UX problem, not an agent reliability problem, fix the input, don't penalize the model.

The judge can share the agent's blind spots. If both run on the same model family, they may agree on a wrong interpretation in lockstep, inflating your scores. Use a different model for judging where you can, and keep a human gold set as the ground truth the judge is forever measured against.

And users' intents shift mid-conversation. The thing they meant in turn one isn't what they mean by turn five. Evaluate intent satisfaction at the level of the resolved goal at the end of the interaction, not the literal opening request, otherwise you'll grade the agent for failing to do something the user themselves abandoned.

Insights Most People Overlook

The clarification question is a feature you should be scoring, not an interruption you should be minimizing. Most teams optimize agents toward fewer questions because questions feel like friction. But on under-specified, high-stakes tasks, a well-timed clarifying question is the single strongest predictor that the agent will do what the user meant. Measure clarification appropriateness, and you'll often find your most "autonomous-looking" agent is actually your least trustworthy one, it just guesses confidently instead of asking.

Your best intent-failure detector is already in your logs, unlabeled, for free. Immediate user corrections, re-issued instructions, and undo actions are ground-truth intent-miss signals that cost nothing to collect and don't require a survey or an LLM judge. Most teams instrument elaborate eval pipelines while ignoring the correction events streaming past them in production. Mine those first.

"Did what the user meant" and "did what the user asked" should be two separate columns in your eval, forever. The instinct is to collapse them into one success metric. Don't. Keeping them split is what surfaces the literal-but-wrong failure mode at all, the moment you merge them, every confidently-wrong completion gets counted as a win and the most dangerous failure class becomes invisible.

Inter-annotator disagreement is a measurement, not a nuisance. When two human reviewers split on whether an agent did what the user meant, the natural reaction is to "resolve" it and move on. The far more valuable move is to log the disagreement rate as a metric. High disagreement on a task type means the intent itself is genuinely ambiguous, which tells you that no agent, however good, can score well there, and the fix lives in your product's input design, not your model.

Per-outcome pricing forces intent measurement whether you're ready or not. The shift toward charging per resolved outcome rather than per task quietly changes what you must measure. Under per-task billing you can get away with measuring completion. The moment you bill per outcome, "did the agent do what the user meant" stops being a quality nicety and becomes the definition of the unit you're selling, and an inaccurate intent metric becomes a direct revenue leak.

References

• Building Effective Agents, Anthropic Engineering
• Getting Started With OpenAI Evals, OpenAI Cookbook