The Token-Volatility Problem: Budgeting When Inference Costs Swing Week to Week

Table of Contents

• What Token Volatility Actually Means
• Where the Swings Come From
• Why a Falling Per-Token Price Doesn't Calm Your Bill
• The Budgeting Problem in Concrete Numbers
• A Practical Framework for Volatile Inference Budgets
• Pricing and Contract Moves That Absorb the Swings
• Instrumentation: What You Have to Measure
• Insights Most People Overlook
• References

What Token Volatility Actually Means

When people in the agent business say "inference is getting cheaper," they're usually quoting a list price: dollars per million input and output tokens, published by a model provider. That number does trend down over time, and dramatically. But the list price is the least volatile thing in the whole equation. It's the only stable input you have.

Token volatility is the week-to-week variance in what it actually costs you to complete a unit of work. Not the price of a token, the cost of a task. And that number can move 30 or 40 percent in either direction across a single billing cycle without anyone touching the price sheet. A coding agent that averaged 180,000 tokens per resolved ticket in March can average 320,000 in April because the underlying model started "thinking" more, or because your customers' codebases got messier, or because a prompt change you shipped on a Tuesday added a verification pass nobody fully costed.

That's the distinction that trips up finance teams. They model COGS as tokens × price, treat price as the variable to watch, and discover too late that the token count is the thing thrashing around. The price is a published constant for weeks at a stretch. The consumption is a living, breathing function of model behavior, customer behavior, and your own orchestration code.

Where the Swings Come From

There isn't one source of volatility. There are at least five, and they rarely move in the same direction at the same time, which is exactly why the aggregate looks so noisy.

Model version churn. Providers ship new model versions constantly, and a "minor" update can change token economics overnight. A model tuned to reason more before answering will burn more output tokens, sometimes 2-3x, on the same task while improving accuracy. You didn't change anything. Your bill changed anyway. Reasoning models made this far worse: the explosion in "thinking" tokens means a chunk of your spend now goes to tokens the customer never sees and can't be shown.

Provider repricing and capacity tiers. List prices fall, but the terms underneath shift too: rate limits, batch discounts, priority-tier surcharges, and the availability of cheaper context-caching paths all move. If you got rate-limited during a traffic spike and your fallback routed traffic to a pricier model or a pricier tier, your effective rate jumped for that window even though no headline price changed.

Retry and fan-out behavior. This is the big, under-modeled one. A single task that nominally costs three model calls can balloon into fifty when tool calls fail, the agent loops, or sub-agents get spawned. Retries are stochastic, they spike when an upstream API is flaky or a model version regresses on a particular task type. A bad week for one of your tool dependencies is a bad week for your token bill. (This connects directly to the hidden-cost-of-retries problem, where one task quietly becomes dozens of calls.)

Customer mix and workload drift. Your cost-per-task is a blended average across every customer and every job. Onboard one enterprise customer with gnarly, high-context workloads and your blended token consumption climbs even if every existing customer is unchanged. Usage is lumpy and the mix reweights weekly.

Your own shipping cadence. Every prompt tweak, every added guardrail, every new verification step you deploy changes the token footprint of a task. Teams ship these constantly and rarely cost them before merge. The volatility you blame on "the model" is frequently self-inflicted.

Why a Falling Per-Token Price Doesn't Calm Your Bill

Here's the uncomfortable truth that catches founders off guard: aggregate token prices have fallen something like 10x year over year for equivalent capability, and almost nobody's agent bill went down. This is the heart of the so-called Jevons paradox in AI inference, when a resource gets cheaper, you consume so much more of it that total spend rises.

Cheaper tokens don't reduce your bill because you immediately spend the savings on capability. The moment a token costs half as much, the rational move is to let the agent reason twice as long, run an extra verification pass, retrieve more context, or call a second model to check the first. Every one of those is a quality improvement your customers reward, and every one of them eats the price decline. The savings get reinvested into autonomy and reliability before they ever reach gross margin.

So the falling price line lulls you into the wrong mental model. You budget assuming costs decay smoothly downward, and instead they ratchet around a roughly flat or rising trend with sharp weekly variance on top. The price decline is real but it's a tailwind you spend, not a floor you can bank on. Anyone budgeting on "it'll be cheaper next quarter" is building a forecast on a number they will choose to disprove.

The Budgeting Problem in Concrete Numbers

Make it tangible. Say you sell a support-resolution agent at a flat $4 per resolved ticket. Your modeled cost is 200,000 tokens per ticket at a blended $6 per million tokens, so $1.20 of COGS, a clean 70 percent gross margin on the inference line. Looks healthy.

Now run four weeks:

• Week 1: model version update adds a reasoning pass. Tokens per ticket climb to 260,000. Cost $1.56. Margin drops to 61%.
• Week 2: an upstream CRM API gets flaky, retries spike, effective tokens hit 340,000 on the affected subset. Blended cost $1.74. Margin 56%.
• Week 3: things stabilize, you ship a caching improvement, tokens fall to 190,000. Cost $1.14. Margin 71%.
• Week 4: you onboard an enterprise account with long ticket histories; blended consumption rises to 250,000. Cost $1.50. Margin 62%.

Nothing catastrophic happened. No outage, no pricing change from the provider. Yet your gross margin on that product swung between 56 and 71 percent inside a month, and your average for the month, call it 62 percent, is meaningfully below the 70 you pitched the board. If you'd priced thinner, say at $2 a ticket, week 2 would have put you underwater on every resolution.

That's the budgeting problem in one table. Per-task pricing converts cost volatility directly into margin volatility, and usage-based forecasting is genuinely hard precisely because the consumption per unit isn't stable. You can't average your way out of it if you priced against the optimistic week.

A Practical Framework for Volatile Inference Budgets

You don't solve volatility by predicting it perfectly. You solve it by budgeting in a way that's robust to being wrong. A few moves that actually hold up:

Budget to the P75, not the mean

Stop forecasting against your average cost-per-task. Pull the distribution of weekly cost-per-task over the last quarter and budget to the 75th percentile, the cost level you exceed only one week in four. Price and plan margin against that number. Yes, it looks more expensive on paper. It also means a normal bad week doesn't blow your forecast, and a genuinely abnormal week is your only surprise. Budgeting to the mean guarantees you're under-provisioned half the time.

Hold an inference reserve

Treat token spend like a cloud bill with a buffer, not a fixed line. Carve out an explicit reserve, 15 to 25 percent of modeled inference COGS, that absorbs bad weeks without triggering a fire drill. When a week comes in under budget, the reserve refills. This is ordinary financial hygiene that AI teams skip because they think of tokens as deterministic. They aren't.

Decompose cost-per-task into drivers you can watch

A single blended cost-per-task number tells you that something moved but not what. Break it into: tokens per successful task, retry multiplier, blended price per million, and cache-hit rate. Now when the aggregate jumps you can tell whether it was the model getting chattier (tokens up), an upstream dependency failing (retries up), a routing change (price up), or a regression in caching (hit rate down). You can't manage a number you can't decompose.

Set per-task and per-customer spend ceilings

The runaway agent is the tail risk that turns a bad week into a bad quarter. A loop that should cost forty cents and instead burns eleven dollars before a human notices is a real failure mode. Hard caps, kill the run at N model calls or $X spend, convert an unbounded tail into a bounded, budgetable one. You'd rather fail a task loudly than complete it at a 400 percent loss.

Re-baseline on every model migration

Treat each model version change as a planned re-budgeting event, not a silent swap. Before you migrate production traffic, run a representative sample through the new version and measure the token delta. A version that's 5 percent more accurate but 60 percent more expensive per task is a pricing decision, not an engineering one, and it needs to reach whoever owns margin before it ships.

Pricing and Contract Moves That Absorb the Swings

Budgeting controls the cost side. Pricing decides who eats the variance. The two have to be designed together.

Don't pure pass-through. The tempting move is to bill the customer for whatever tokens their tasks consumed, plus a markup, so volatility becomes their problem. It rarely works. Customers hate consumption pricing they can't predict, finance teams reject line items that swing 40 percent month to month, and "we'll just pass through model costs" is a known margin trap, pass-through transfers the volatility but also transfers the blame, and it makes your own revenue as lumpy and unforecastable as your costs.

Price per outcome, hedge with a floor. Per-outcome pricing (per resolved ticket, per qualified lead, per merged PR) is what customers actually want to buy, because it maps to value. But it puts 100 percent of the cost volatility on you. The fix is a blended structure: a committed minimum or platform fee that covers your fixed inference reserve, plus per-outcome pricing on top. The floor smooths your revenue against their lumpy usage and gives you a budgetable base.

Build volume tiers around variable cost, not fixed assumptions. Standard SaaS volume discounts assume marginal cost approaches zero at scale. Yours doesn't, your COGS scales close to linearly with usage. Discounts that ignore that will price your best weeks into losses at your worst customers' volumes. Tier the discount to your variable cost curve, and reserve the deepest tiers for workloads you can serve cheaply (cacheable, short-context, open-weight-eligible).

Use the model portfolio as a margin lever. You don't have to serve every task on the frontier model. Routing the easy 70 percent of tasks to a cheaper or open-weight model and reserving the frontier model for the hard tail is one of the strongest stabilizers you have, it both lowers the mean and compresses the variance, because the cheap model's behavior is the part of your traffic you control most tightly. McKinsey's work on capturing value from AI keeps landing on the same point: the economics are won in deployment discipline, not model selection alone.

Instrumentation: What You Have to Measure

None of the above works without per-task cost telemetry, and most teams discover they don't have it until a bad month forces the question. The non-negotiables:

• Per-task token attribution. Every task gets a cost record: input tokens, output tokens, thinking tokens, tool-call tokens, model version used, retry count. Aggregated bills from the provider are useless for diagnosing volatility, you need the cost stamped on the unit of work.
• A weekly cost-per-task distribution, not just an average. Track the spread. A stable mean hiding a widening tail is an early warning that your runaway scenarios are getting more common.
• Cost-anomaly alerting. A spend spike should page someone the day it happens, not surface in the monthly close. Alert on cost-per-task crossing your P90, on retry multipliers above threshold, and on any single run exceeding its cap.
• Margin-per-outcome, refreshed weekly. The number that actually matters, revenue per outcome minus fully-loaded inference cost per outcome, should be a live dashboard metric, not a quarterly reconstruction. If you can't see margin compressing in week two, you'll find out in the quarterly when it's a trend instead of a blip.

Get that instrumentation in place and token volatility stops being a mystery. It becomes a measured, bounded, budgetable variable, still annoying, but no longer dangerous.

Insights Most People Overlook

Volatility, not the price level, is what kills thin-margin agents. Two vendors with identical average COGS can have wildly different survival odds. The one with tight variance can price close to cost and live; the one with fat-tailed weekly swings gets killed by the bad weeks even though its average looks fine. The industry obsesses over lowering the mean. The teams that last are the ones that compressed the variance, through caching, routing, and hard caps, so they could price near the average without getting wiped out by the tail.

Your most loyal customers often drive the worst volatility. Expansion is supposed to be good news, but in usage-based agent products, your power users are exactly the accounts running the longest-context, most-retry-prone, most-expensive workloads. The customer who loves the product most can be the one quietly destroying your margin. Net revenue retention can look spectacular while gross margin silently bleeds, you have to watch both, per cohort, or you'll celebrate the wrong thing.

Reasoning models inverted the old cost intuition. For years the rule was "output tokens cost more than input, so keep responses short." Reasoning models broke that: the expensive part is now invisible thinking the customer never sees, and it scales with problem difficulty in ways you can't cap by trimming the visible answer. A budget built on the old intuition under-provisions for exactly the hard tasks that matter most.

The cheapest week is the most dangerous one for pricing decisions. When caching and routing line up and a week comes in unusually cheap, there's enormous pressure to cut prices or deepen discounts to win deals. Don't price off your best week. The conditions that made it cheap, high cache-hit rates, easy workload mix, a stable model version, will revert, and you'll have locked in a price against a cost floor you can't hold.

"Pass it through" doesn't escape volatility; it relocates the complaint. Vendors who pass token costs straight to customers think they've offloaded the problem. They've actually signed up to defend a line item that swings 40 percent a month to a finance team that hates surprises, which generates churn and support load that often costs more than just eating the variance behind a predictable price would have.

References

• Anthropic, Claude 3.7 Sonnet and extended thinking
• Andreessen Horowitz, The Economics of Generative AI
• Bessemer Venture Partners, Scaling with Usage-Based Pricing
• McKinsey & Company, The State of AI