The True Cost of an Agent's "Thinking" Tokens (And Why Your Margin Model Is Probably Wrong)

Table of Contents

• What "Thinking" Tokens Actually Are
• Why Thinking Tokens Are Priced Like Output, Not Input
• The Attribution Problem: You Pay For Tokens You Cannot Read
• Running the Real Numbers on a Single Task
• Where the Cost Hides in an Agent Loop
• The Levers That Actually Move Thinking Cost
• How to Price When Thinking Is Variable and Invisible
• Insights Most People Overlook
• References

What "Thinking" Tokens Actually Are

A few years ago, a model gave you an answer. Now, the better ones think first.

When a reasoning model, OpenAI's o-series, Anthropic's extended-thinking Claude, Google's Gemini thinking variants, DeepSeek-R1, receives a hard prompt, it generates a long internal monologue before it commits to a response. It works through the problem, second-guesses itself, tries an approach, abandons it, tries another. That internal monologue is made of tokens. Real tokens. Tokens you pay for at the output rate, even though the end user never sees a single one of them.

This is the part most cost models miss entirely. People budget for the prompt going in and the answer coming out. The thinking is a third bucket sitting between them, and on hard tasks it's frequently the biggest bucket. A 200-token question can trigger 4,000 tokens of reasoning that collapse into a 300-token answer. You were billed for roughly 4,300 output-rate tokens and you can show the customer 300 of them.

For a chatbot, this is an annoyance. For an autonomous agent that runs dozens of model calls per task, each one potentially reasoning, it's a structural cost problem, and it's the kind of cost problem that doesn't show up until you reconcile the invoice at the end of the month and find your "cost-per-completed-task" is double what your spreadsheet said.

Why Thinking Tokens Are Priced Like Output, Not Input

Here's the asymmetry that makes this expensive. Input tokens, your prompt, your context, your retrieved documents, are cheap. Output tokens, the stuff the model generates, cost several times more. Across the major frontier providers, output is commonly priced at three to five times the input rate, and reasoning tokens are generated, so they bill at that higher output rate.

That pricing makes sense from the provider's side: generation is sequential and compute-heavy in a way that reading context isn't. But it means the most invisible part of an agent's work is also its most expensive part per token. The provider documentation is explicit about this, Anthropic's own guidance on extended thinking notes that thinking tokens are billed as output tokens and count toward your generation costs.

So you have a category of token that is (a) billed at the premium rate, (b) generated in large and unpredictable volumes, and (c) invisible to the end user, which means you can't even use it as a selling point. It's pure cost-of-goods with no narrative value. If you're running a GaaS business, that's the worst possible profile for a line item, and it's exactly why falling headline token prices haven't lowered agent bills the way operators expected.

The Attribution Problem: You Pay For Tokens You Cannot Read

There's a second wrinkle that turns a cost problem into a forecasting nightmare: you often can't see the thinking tokens.

For safety and competitive reasons, several providers don't return the raw reasoning trace. OpenAI's reasoning models, for instance, expose a summary of the thinking and a token count, but not the full verbatim chain, and you're billed on that count. So your COGS for a task is partly determined by a process you can audit only as an aggregate number after the fact. You can't read it, you can't easily predict it, and you can't trivially cap it without degrading quality.

This breaks a lot of naive unit-economics models. If your pricing assumes "average task = 5,000 tokens," but reasoning effort scales with problem difficulty, and the hard 10% of tasks burn 40,000 reasoning tokens each, your blended cost is dominated by a tail you didn't budget for. This is the same tail-risk dynamic that shows up in the hidden cost of retries and in worst-case runaway-spend modeling; thinking tokens just make the tail fatter and harder to see.

The practical consequence: any GaaS operator on per-task or per-outcome pricing needs reasoning-token consumption as a tracked, dashboarded metric, broken out from input and visible output. If you're only watching total spend, you're flying with one instrument.

Running the Real Numbers on a Single Task

Let's make this concrete, because hand-waving about "more tokens" doesn't help anyone price a product.

Take a mid-complexity agent task: triage a customer support ticket, look up two records via tool calls, decide on a resolution, and draft a reply. Suppose this runs on a reasoning model priced at $3 per million input tokens and $15 per million output tokens, roughly representative of mid-tier frontier reasoning pricing in 2026.

• Input: ~8,000 tokens (system prompt, tool schemas, retrieved context, conversation history). Cost: 8,000 × $3/M = $0.024.
• Visible output: ~400 tokens (the drafted reply plus structured fields). Cost: 400 × $15/M = $0.006.
• Thinking tokens: ~6,000 tokens across the reasoning steps. Cost: 6,000 × $15/M = $0.090.

The thinking alone is $0.090, three-quarters of the total $0.12 task cost, and fifteen times the cost of the answer the customer actually receives. Now multiply by the reality that this agent loops: it reasons before each tool call, reasons again to interpret the tool result, reasons once more to compose the reply. The 6,000-token figure is conservative for a multi-step loop; a genuinely autonomous agent can easily triple it.

If you priced this task at $0.10 to the customer because your back-of-envelope said "tokens are basically free now," you're losing money on every run and you won't know until reconciliation. This is precisely why cost-per-completed-task has to be measured on real invoices, not projected from token list prices.

The uncomfortable part: as model providers ship "smarter" reasoning, the default thinking budgets tend to increase, not decrease. Better answers, more expensive thinking. The trend line for quality and the trend line for your COGS point the same direction.

Where the Cost Hides in an Agent Loop

Single-call math undersells the problem because agents don't make single calls. The compounding happens in four places, and each one multiplies thinking spend.

Per-step reasoning. Every node in an agent's plan-act-observe loop can trigger fresh reasoning. A ten-step task isn't one thinking pass; it's up to ten, each re-reasoning over an ever-growing context.

Re-reading the transcript. As the agent accumulates history, each new step re-ingests the whole conversation as input. The input bucket grows linearly, and the model reasons over a larger and larger blob each turn. Thinking cost and input cost climb together.

Sub-agent fan-out. When an orchestrator spawns sub-agents, each sub-agent does its own thinking. A coordinator reasoning about how to delegate, plus three workers each reasoning about their slice, is four reasoning streams where your model assumed one. This is the fan-out economics problem viewed through the thinking-token lens, and it's where margins go to die quietly.

Failed branches and retries. When a step fails and the agent re-plans, the reasoning it already paid for is sunk. You bought 5,000 tokens of thinking that led to a dead end, and now you're buying 5,000 more to recover. Retries don't just double the visible calls; they double the invisible thinking behind them.

Put together, a task your spreadsheet modeled as "one reasoning call" can carry six or eight reasoning passes in production. That's not an edge case. That's a Tuesday.

The Levers That Actually Move Thinking Cost

The good news: thinking cost is more controllable than retry cost or token-price volatility, because you have direct knobs. The bad news: most of them trade against quality, so you're tuning a dial, not flipping a switch.

Reasoning effort settings. The major providers now expose effort or thinking-budget controls, low/medium/high, or an explicit token cap on the reasoning. The single highest-leverage move in GaaS cost control is matching effort to task difficulty instead of running everything at maximum. Most tasks an agent handles are easy. Reserve deep reasoning for the genuinely hard ones, and route the rest to a lower budget or a non-reasoning model entirely.

Model routing. Not every step needs a frontier reasoner. Classification, formatting, and simple extraction can run on a cheap, fast, non-thinking model; only the genuinely hard planning steps escalate. A well-built router can cut reasoning spend by half or more without users noticing, and it's the practical core of the open-weight vs. frontier model economics decision.

Prompt caching. Since input grows every step, caching the stable parts of the context, system prompt, tool schemas, static reference material, cuts the input cost dramatically and, on some providers, reduces the work the model redoes each turn. McKinsey's analysis of scaling generative AI from pilot to production repeatedly lands on the same point: the cost killers in production are architectural, not per-token, and caching is one of the biggest architectural levers available.

Capping the loop. A hard ceiling on steps and a thinking-token budget per task turns an unbounded tail risk into a bounded one. You'd rather an agent fail cleanly at step 12 and hand off to a human than reason its way to $4 of spend on a task you priced at $0.30. This is the same instinct behind why some agent startups are quietly capping autonomy to protect margin.

None of these are free. Lower effort means occasionally worse answers; aggressive routing means occasional misroutes. The job isn't to minimize thinking cost, it's to spend it where it changes the outcome and starve it where it doesn't.

How to Price When Thinking Is Variable and Invisible

If thinking cost is variable, premium-rate, and partly unobservable, what does a defensible pricing model look like?

Start by refusing to pass through raw token costs. The margin trap of "we'll just pass through model costs" is real, and thinking tokens make it lethal: you'd be exposing your customer to a cost line neither of you can predict or audit. Pass-through pricing on reasoning tokens is volatility laundering, and the customer will hate the unpredictable bill even more than you hate the unpredictable cost.

Instead, price on the outcome and self-insure the variance. Charge per completed task or per verified result, build your price off the blended median cost across your real task distribution, not the average, which the fat tail distorts, and carry a margin buffer sized to the tail. You're effectively selling the customer a fixed price and absorbing the reasoning-token volatility yourself, which is only sane if you've measured the distribution and capped the worst case.

The two non-negotiables: instrument reasoning tokens as a first-class metric, and cap them per task. Without the instrumentation you're guessing; without the cap your worst-case task is unbounded and one pathological run can erase the margin from a hundred clean ones. Everything else, routing, caching, effort tuning, is optimization on top of those two foundations.

Insights Most People Overlook

Cheaper models can cost more, not less. A weaker reasoning model often thinks longer to reach the same answer a stronger model reaches quickly. If you switch to a "cheaper" model and its per-token price is 40% lower but it generates 3x the reasoning tokens, your task got more expensive. The right comparison is never per-token price, it's cost-per-completed-task, including all the thinking. Teams that benchmark on list price routinely pick the more expensive option.

Thinking tokens are a privacy and compliance surface, not just a cost. That hidden reasoning trace can contain a verbatim restatement of sensitive input, PII, secrets, internal logic, sitting in the provider's generation path. When you can't read the trace, you also can't easily audit what it contains. For regulated verticals, "we pay for tokens we can't inspect" is a security finding, not just a margin note.

The "smarter model" upgrade is often a stealth price increase. When a provider ships a new model that reasons more by default, your per-task cost rises even if the headline per-token price held steady or dropped. Plenty of operators "upgraded" and watched COGS climb without changing a line of their own code. Always re-benchmark thinking-token consumption on a model change, the list price tells you almost nothing.

Most of your thinking budget is spent on tasks that didn't need it. Task difficulty in production is heavily skewed: a large majority of runs are easy, and a small minority are genuinely hard. If you run uniform reasoning effort, you're spending premium thinking on the easy majority for no quality gain. The single biggest waste in most GaaS cost structures isn't the hard tasks burning tokens, it's the easy tasks burning them needlessly.

You can't optimize what you bill as one number. Vendors who report only "total model spend" have no idea what fraction is thinking versus input versus visible output. Splitting that one number into three is the cheapest, highest-return instrumentation you can build, and most operators haven't done it. The split is the insight.

References

• Anthropic, Building with Extended Thinking (Claude documentation)
• OpenAI, Reasoning Models Guide
• McKinsey & Company, QuantumBlack: Scaling Generative AI Insights