Inference Providers Are Fighting for Agent Workloads, and the Old Pricing Math No Longer Works

Table of Contents

• Why Agent Inference Is a Different Animal
• The Contenders and Where They're Placing Bets
• The Frontier Labs
• The Speed Specialists
• The Open-Model Platforms
• The Hyperscalers
• The Five Axes Agent Builders Actually Buy On
• The Economics: Why Per-Token Pricing Lies to You
• Prompt Caching Is the Real Battlefield
• How to Choose Without Getting Locked In
• Insights Most People Overlook
• References

Why Agent Inference Is a Different Animal

Spend a week instrumenting a real agent and you stop thinking about inference as "the API call." You start thinking about it as a population of calls with wildly uneven shapes.

A consumer chatbot turn is roughly one round trip: a few thousand tokens in, a few hundred out, and a human waiting who will tolerate a couple of seconds. An agent task is the opposite of tidy. A mid-complexity workflow, say, triaging a support ticket and drafting a resolution, might issue fifteen to forty model calls. Some are tiny routing decisions. Some drag a 60,000-token context window of accumulated tool output behind them. A few are reasoning-heavy. And critically, most of them are not waited on by a human at all; they're waited on by the next step in the agent, which means latency compounds instead of hides.

That compounding is the whole story. If a single inference call has a p99 latency of four seconds and your agent chains thirty of them, your worst-case task tail isn't four seconds, it's a structural problem that shows up as timeouts, abandoned runs, and angry customers. Chatbot buyers optimize median latency because a human reads at human speed. Agent buyers optimize tail latency because machines don't forgive a slow straggler in the chain. This is exactly why the observability stack for agent infrastructure and inference selection are tied at the hip: you can't choose a provider intelligently until you can see your own latency distribution.

The second difference is repetition. Agents reuse the same system prompt, the same tool schemas, and the same retrieved context across call after call within one task. A chatbot rarely gets to amortize that. An agent does it constantly, which makes prompt caching not a nice-to-have but the single biggest lever on cost. More on that below.

The third difference is failure semantics. When a chatbot call fails, a human retries or rephrases. When an agent call fails mid-chain, you need automated retries, fallbacks, and circuit breakers, and the inference provider's error behavior, rate-limit headers, and 5xx rate become part of your reliability budget. This is why provider choice bleeds directly into reliability infrastructure: retries, fallbacks, circuit breakers.

The Contenders and Where They're Placing Bets

The market sorted itself into four camps, each optimizing for a different thing. Knowing which camp a provider belongs to tells you more than any benchmark.

The Frontier Labs

OpenAI and Anthropic sell the models everyone actually wants to run agents on, and they've been racing to make their own first-party inference attractive for autonomous workloads specifically. The tell is in the features: prompt caching, batch APIs, priority-tier latency, and structured tool-calling that's reliable enough to chain. Anthropic's published guidance on prompt caching and token-efficient tool use reads like a document written by people who watched a lot of agents burn money on repeated context. The frontier labs win on raw capability and tool-calling reliability. They lose on price ceilings and on the discomfort of single-vendor dependence.

The Speed Specialists

Groq and Cerebras took a different swing: custom silicon built to make token generation absurdly fast. For agents, the pitch is seductive because latency compounds, so shaving per-call time pays off thirty times over in a thirty-call chain. The catch is model selection. Speed specialists run a curated set of open-weight models, not the frontier proprietary ones, so you're trading peak reasoning quality for throughput. For agents whose calls are mostly routing, extraction, and classification rather than deep reasoning, that trade is often correct, and it pairs naturally with the model-routing layer: cheap model when you can.

The Open-Model Platforms

Together AI, Fireworks AI, Baseten, and Fdric-style serving platforms compete to run Llama, Qwen, DeepSeek, Mistral, and the rest of the open-weight catalog at production quality. This is the most price-competitive corner of the market, and it's where per-outcome GaaS pricing actually pencils out, because you can fine-tune, quantize, and pin models without a frontier-lab markup. These platforms increasingly ship agent-specific features: structured output enforcement, speculative decoding, and dedicated deployments that protect you from noisy-neighbor latency. The risk is operational, you own more of the reliability story, which connects to the self-hosted vs. managed agent infrastructure decision.

The Hyperscalers

AWS Bedrock, Google Vertex, and Azure AI sell inference as one more service inside the cloud you already buy. Their advantage isn't price or speed; it's procurement, data residency, and the fact that your data never leaves the VPC. For regulated buyers, that's decisive. The cost is that you're often a release cycle or two behind on the newest models and the sharpest caching features.

The Five Axes Agent Builders Actually Buy On

Strip away the marketing and providers compete on five things that matter for autonomous workloads, roughly in this order:

1. Tail latency under chaining. Not median, p95 and p99, because that's what determines whether a long task finishes. Ask for latency distributions, not averages, and benchmark on your own prompt shapes.
2. Prompt-cache economics. What's the cache write premium, the cache read discount, and the TTL? A five-minute TTL is useless for a slow-running agent; a one-hour TTL changes your whole cost model.
3. Tool-calling reliability. How often does the model emit malformed tool calls or hallucinate arguments under load? This is infrastructure-layer reliability, and it ties directly to tool-calling reliability at the infrastructure layer.
4. Throughput and rate limits. Agents are bursty. A provider with generous tokens-per-minute and graceful 429 backpressure beats one with a higher headline speed but a stingy quota.
5. Model breadth and portability. Can you route across models, swap providers, and avoid lock-in? The more your agent depends on one proprietary model's exact behavior, the more a price hike or deprecation can wreck you.

Notice that price isn't first. For most production agents, a provider that's 20% cheaper per token but adds two seconds of tail latency and a worse cache loses on total cost of ownership once you account for failed and retried runs.

The Economics: Why Per-Token Pricing Lies to You

Here's the uncomfortable truth GaaS operators learn the hard way: the per-million-token price on the pricing page is nearly meaningless for predicting your agent's bill.

Three things distort it. First, output tokens usually cost three to five times what input tokens cost, and agents that reason or write generate a lot of output, so a model with cheap input and expensive output can be more costly than its headline suggests. Second, the cache hit rate dominates everything. An agent that reuses a 50,000-token system-and-context block across thirty calls pays full freight once and a 90%-discounted rate twenty-nine times, if the provider's caching is good and your call pattern keeps the cache warm. Get that wrong and the same workload costs ten times more. Third, reasoning models bill for hidden "thinking" tokens you never see in the output, which can quietly double the real token count per call.

a16z's analysis of the falling cost of LLM inference captures the macro trend, raw token prices are collapsing year over year, but that decline masks a divergence at the agent layer: the providers winning agent workloads aren't the cheapest per token, they're the ones whose caching, batching, and tail-latency behavior make a chained workload cheap. Two providers with identical sticker prices can produce 5x different bills on the same agent.

The practical move: never benchmark on price-per-token. Run your actual agent against two or three providers for a day, measure cost-per-completed-task and p95 task latency, and decide on those. Everything else is a vanity number. This is the heart of the GaaS infrastructure cost stack, decomposed.

Prompt Caching Is the Real Battlefield

If I had to name the single feature that's reshaping the inference-for-agents competition, it's prompt caching, and it's underappreciated because chatbot buyers barely use it.

The mechanics: the provider stores the key-value attention state for a prefix of your prompt, so the next call that shares that prefix skips recomputing it. For agents, the shared prefix is huge, your system prompt, tool definitions, few-shot examples, and accumulated context all repeat across the calls in a task. Cache that, and you cut both cost (cached reads are typically 75-90% cheaper) and latency (less prefill to compute) on every call after the first.

But the implementations differ in ways that matter enormously:

• Explicit vs. automatic caching. Some providers make you mark cache breakpoints; others detect repeated prefixes automatically. Explicit gives you control but demands engineering; automatic is convenient but can miss your hot path.
• TTL. Cache lifetimes range from a few minutes to an hour or more. A long-running agent with a five-minute TTL will keep paying cold-cache prices; the difference is real money.
• Write premiums. Some providers charge extra to write to the cache, which only pays off if you read it enough times, fine for agents, but it changes the break-even math.

The deeper point connects to the context-window economy: managing what agents remember: how you structure an agent's context determines how cacheable it is. Put the stable stuff (system prompt, tools) first and the volatile stuff (latest tool output) last, and your cache hit rate climbs. Interleave them carelessly and you blow the cache on every call. Inference provider choice and context engineering are the same optimization viewed from two sides.

How to Choose Without Getting Locked In

The strategic answer is to not marry one provider. Build an abstraction, a thin routing layer, so your agent calls "a model" rather than "OpenAI's model," and you can shift workloads as prices, latency, and capabilities move, which they do monthly.

A pragmatic posture for a GaaS operator:

• Route by call type. Send cheap routing and extraction calls to a fast open-model provider; send the hard reasoning calls to a frontier lab. A multi-provider router captures most of the cost savings without sacrificing quality where it counts.
• Benchmark on your workload, refresh quarterly. The leaderboard changes fast. A provider that was 30% behind on latency last quarter may have shipped new silicon or a better cache. Re-run your cost-per-task benchmark on a schedule.
• Negotiate caching and rate limits, not just price. At volume, the levers that actually move your economics are cache TTL, committed throughput, and priority latency tiers, not the per-token rate.
• Keep a fallback warm. Provider outages happen. A circuit breaker that fails over to a second provider keeps your agents running when one goes down, and it gives you negotiating leverage besides.

The meta-lesson: inference is becoming the most contested layer of the agent stack precisely because it's where cost, latency, and reliability all converge. Treat provider selection as an ongoing operational discipline, not a one-time procurement decision, and revisit it as part of your broader picks-and-shovels map of agent infrastructure.

Insights Most People Overlook

1. The cheapest provider per token is often the most expensive per completed task. Because agents retry failed calls and abandon timed-out runs, a provider with a slightly higher token price but a tighter latency distribution and lower error rate frequently produces a lower bill once you count the waste. Nobody puts "cost per successful task" on a pricing page, but it's the only number that matters.

2. Reasoning models quietly break per-token budgets. Hidden "thinking" tokens, billed but never shown, mean a reasoning model's real token consumption per call can be 2-3x what the visible output implies. Agents that lean on reasoning for planning can blow a budget that looked fine in a spreadsheet built on visible tokens alone.

3. Cache TTL is a stealth lock-in mechanism. Providers with long cache lifetimes make your economics depend on keeping traffic warm with them. The moment you split traffic across providers, your hit rate drops on both, and the per-token math that justified multi-provider routing partially unwinds. The portability benefit and the caching benefit are in genuine tension, and almost nobody models that tradeoff explicitly.

4. Speed specialists win more agent workloads than benchmarks suggest. Because most calls in a real agent chain are mundane, routing, extraction, classification, the marginal quality of a frontier model is wasted on them while its latency tax compounds across the chain. A "weaker but faster" model on custom silicon often produces better end-to-end task outcomes simply by finishing the chain faster and more predictably.

5. The inference provider is part of your security perimeter, whether you acknowledge it or not. Every tool output and piece of retrieved context an agent sends to inference passes through that provider. For autonomous systems handling sensitive data, the provider's data-handling, retention, and isolation guarantees are a security decision disguised as a procurement one, which is why hyperscaler in-VPC inference wins regulated buyers even at a latency and feature cost.

References

• Anthropic, Prompt Caching and Token-Efficient Tool Use Documentation
• a16z, Welcome to LLMflation: The Falling Cost of LLM Inference