The GaaS Infrastructure Cost Stack, Decomposed: Where the Money Actually Goes

Table of Contents

• Why The Cost Stack Matters More In GaaS Than In SaaS
• Layer 1: Inference (The Part Everyone Sees)
• Layer 2: Orchestration And Runtime
• Layer 3: Memory, State, And Retrieval
• Layer 4: Tools, Integrations, And The Outbound API Tax
• Layer 5: Observability, Eval, And Reliability
• Layer 6: The Human Layer Nobody Puts On The Slide
• A Worked Example: One Support Agent Task, Fully Costed
• Insights Most People Overlook
• References

Why The Cost Stack Matters More In GaaS Than In SaaS

In classic SaaS, your marginal cost of serving one more user rounds to zero. You write the software once, and the hundred-thousandth seat costs you a rounding error in compute. That economic shape is why software trades at the multiples it does, and it's the mental model most agent founders quietly imported when they started selling agents.

It does not hold for GaaS. Every single agent run burns variable cost: tokens, vector lookups, tool calls, sandbox seconds, sometimes a human review. When you price per task or per outcome, you've signed up for a cost-of-goods-sold business wearing a software costume. The discipline you actually need looks more like a logistics company watching its cost-per-shipment than a SaaS company watching its CAC payback.

That's the whole reason to decompose the stack. If you can't name where each dollar of a task goes, you can't tell whether your per-task price is a 70% gross margin or a slow bleed. And in agentic systems the bleed is sneaky, because the expensive layers are the ones that don't show up on the model provider's invoice. Let's walk them in order, roughly from most-visible to least.

Layer 1: Inference (The Part Everyone Sees)

Inference is the line item everyone fixates on because it arrives as a clean monthly bill from Anthropic, OpenAI, or your inference provider. It's real, it's large, and it's also the part of the stack that has fallen in price fastest. Frontier-model token prices have dropped by roughly an order of magnitude over an eighteen-month window for equivalent capability, a trend Andreessen Horowitz has documented in its writing on the cost of LLM inference falling and the downstream effect on application economics.

But the headline price-per-million-tokens hides three multipliers that decide your actual inference bill:

Token amplification from agentic loops. A single user-facing "task" is rarely one model call. An agent reasons, calls a tool, reads the result, reasons again, maybe self-critiques, maybe retries. A task that feels like one request can be eight to twenty model invocations, each re-sending a growing context. The dominant cost in a long agent run is usually not the new tokens generated, it's the same context being re-read on every turn. This is exactly the problem the related context-window economy work in this cluster digs into.

The model-tier decision. Running every step on a frontier model is the most common margin mistake in GaaS. Most steps in an agent loop, routing, classification, "did the tool succeed?" checks, don't need a top-tier model. A disciplined routing layer that sends cheap models the cheap work can cut inference cost 50-80% with negligible quality loss. That's a whole sub-topic on its own (model routing), and it's where the difference between a 30% and a 70% gross margin often lives.

Prompt caching. If your agent re-sends the same system prompt, tool definitions, and retrieved documents on every turn, and it does, caching those stable prefixes can knock a large chunk off the per-task bill. Provider-side caching has gone from a niche optimization to table stakes; ignoring it is leaving 50%+ of repeated-context cost on the table.

The takeaway: inference is the visible cost, but it's also the most optimizable one. The teams getting crushed are usually the ones who optimized only this layer and assumed the rest was free.

Layer 2: Orchestration And Runtime

Here's where the invoice stops being legible. Something has to actually run the agent loop, hold the execution state, decide the next step, handle a tool timeout, retry a failed call, and not lose the whole task if a process dies forty minutes in. That something is the agent runtime, and it costs money in ways inference never shows you.

If you self-host orchestration, you're paying for always-on compute that sits mostly idle waiting on model and tool latency. An agent that takes ninety seconds wall-clock might spend eighty of those seconds blocked on I/O, and you're renting the box the whole time. Long-running and async agents make this worse: a task that runs for an hour, or sleeps waiting on a human approval, ties up runtime resources far longer than the few seconds of compute it actually uses.

This is why durable-execution engines and event-driven orchestration have become an infrastructure category of their own, they let an agent pause cheaply instead of holding a hot process. If you buy a managed agent runtime instead, that cost moves onto a vendor invoice (often priced per execution-second or per step), which is cleaner to reason about but rarely cheaper at scale. Either way, orchestration is typically 10-25% of a real task's fully-loaded cost, and almost nobody models it before they ship.

Layer 3: Memory, State, And Retrieval

Agents that do anything useful need to remember and to look things up. Both cost money continuously, not just at task time.

Retrieval. Every RAG query is an embedding call plus a vector-database query plus, often, a re-ranking pass. Individually cheap; in aggregate, not. An agent that retrieves five times per task at scale is running a serious vector-DB workload, and managed vector databases bill on stored vectors, queried vectors, and dimensions, all of which creep upward as your corpus grows.

State and memory. Stateful agents persist conversation history, scratchpad memory, and long-term user memory. That's storage you pay for whether or not the agent runs today, plus the read/write costs each task incurs. Memory-system architecture choices have a direct cost consequence: a design that re-embeds and re-stores aggressively can quietly become one of your larger line items, which is why memory persistence carries a real privacy and cost tradeoff.

The non-obvious bit: retrieval and memory costs scale with your data, not your traffic. You can have flat task volume and a rising infrastructure bill purely because the corpus and the accumulated memory keep growing. SaaS intuition doesn't warn you about this. It's a standing cost that compounds.

Layer 4: Tools, Integrations, And The Outbound API Tax

An agent's value comes from doing things in the real world, and the real world charges admission. Every tool call may hit a paid third-party API, a search API, an enrichment provider, a payments call, a SaaS endpoint with metered access. These pass straight through to your COGS, and they're easy to forget because they don't feel like "AI" cost.

Browser-automation and computer-use agents add a particularly heavy variant: you're now paying for headless browser sessions or virtual-desktop sandbox seconds, which are dramatically more expensive per task than a model call. A computer-use agent clicking through a web app for ten minutes can spend more on the sandbox than on inference.

There's also a reliability tax buried here. Brittle integrations fail, and failed tool calls trigger retries, which means more inference and more tool calls to recover. A 10% tool-failure rate doesn't add 10% to cost; it adds the full cost of the retry loop, including the re-sent context. Reliable tool integration isn't just a quality concern, it's a direct cost lever.

Layer 5: Observability, Eval, And Reliability

You cannot run agents in production blind, which means you're paying for a whole observability stack: tracing every step of every run, logging inputs and outputs, storing traces for debugging, and running evals to catch regressions. Agent traces are large, a single run can produce dozens of spans with full prompt and response payloads, and storing and indexing them at volume is a non-trivial bill of its own.

Then there's evaluation. Quality assurance for agents increasingly means LLM-as-judge passes and replayed test suites, which are themselves inference workloads. If you eval 5% of production traffic with a judge model, you've just added 5% inference cost that produces zero customer-facing output. It's worth it, flying blind on agent reliability is how you lose a customer in one bad run, but it has to be in the model.

Reliability infrastructure (retries, fallbacks, circuit breakers) sits here too, and like the tool layer, its cost is mostly the cost of failure. Gartner has repeatedly flagged that a large share of agentic AI projects stall or get scrapped, and unmodeled reliability and observability cost is part of why the economics don't pencil out in production. Their analysis on agentic AI project cancellations through 2027 is worth reading as a cost-discipline warning, not just a hype check.

Layer 6: The Human Layer Nobody Puts On The Slide

This is the layer that turns a beautiful margin into a mediocre one, and it's almost always missing from cost decks.

Most production agents that touch anything consequential, money, legal text, customer commitments, have a human in the loop somewhere. Maybe a reviewer approves before send. Maybe an ops person handles the 8% of tasks the agent escalates. Maybe an engineer babysits a flaky workflow for the first month of a new customer. That human time is a per-task cost in everything but the accounting, and if you're selling per-outcome, it eats directly into the outcome's margin.

The trap is that human cost scales inversely with agent quality on your hard cases. The easy 80% of tasks the agent nails for pennies; the hard 20% generate the escalations, the corrections, and the support tickets, and those are where the human hours pile up. A provider can look hugely profitable on average task cost while losing money on the long tail, and per-outcome pricing concentrates that risk because you don't get paid more for the hard ones. Infrastructure for human-in-the-loop checkpoints is partly a cost-control discipline: the cheaper and faster you make human review, the more of the long tail you can serve profitably.

A Worked Example: One Support Agent Task, Fully Costed

Let's put rough numbers on a single resolved customer-support ticket to make the stack concrete. These are illustrative, not benchmarks, your mileage varies enormously by workload, but the shape is what matters.

• Inference: 12 model calls across the loop, mixed model tiers, with prompt caching on the system prompt and retrieved docs. Net: roughly $0.06.
• Retrieval + memory: 4 embedding calls, 4 vector queries, plus reading and writing conversation memory. Roughly $0.01.
• Orchestration runtime: ~40 seconds of managed execution across a multi-step durable workflow. Roughly $0.01.
• Tool/API calls: one paid lookup against the order system, one enrichment call. Roughly $0.02.
• Observability + eval: full trace stored, plus a 1-in-10 LLM-judge eval amortized across tasks. Roughly $0.01.
• Human review: 8% of tickets escalate to a human at ~4 minutes each, amortized across all tickets. Roughly $0.05.

Add it up and inference is about $0.06 of an ~$0.16 fully-loaded task, well under half. If you'd priced this agent assuming "cost = the API bill," you'd have undercharged by roughly 2.5x and called your margin healthy right up until the monthly statements arrived. The human and observability layers, the two least-visible ones, together rival the inference cost. That inversion is the entire point of decomposing the stack.

Insights Most People Overlook

1. Your cheapest layer is the one you can't optimize, and your most expensive is the one you control least. Inference keeps getting cheaper and is highly tunable, so it's a shrinking, controllable cost. The human layer barely deflates with model improvements on your hard cases and is the hardest to compress. Over a multi-year horizon, the cost mix of a GaaS business shifts away from the layer everyone optimizes and toward the layer nobody models. Plan your margin curve accordingly.

2. Per-outcome pricing transfers the long-tail cost risk onto you, and the long tail is where the stack is most expensive. Per-task pricing at least lets the hard, multi-retry, human-escalated tasks cost the customer more. Per-outcome flattens that, you eat the variance. It's a great way to win deals and a great way to lose money on exactly the customers whose workflows are messiest. If you sell per-outcome, you need cost telemetry per outcome, not per task, or you're flying blind on the cases that matter most.

3. Reliability spend is negative-cost, not a cost center. A failed tool call or a hallucinated step doesn't just cost the wasted call, it triggers a retry loop that re-sends the entire growing context and may pull in a human. Spending on reliability infrastructure and better integrations reduces total cost by killing the retry-and-escalate cascade. Teams that treat observability and reliability as overhead to be minimized usually have higher all-in cost per successful task than teams that "overspend" on it.

4. Retrieval and memory costs decouple from revenue. Most of the stack scales with task volume, which scales with revenue, healthy. But vector storage, embedding re-indexing, and accumulated long-term memory scale with your corpus and history, which keep growing even on flat traffic. This is the line item most likely to surprise a CFO twelve months in: usage didn't grow, but the bill did. Build a retention and re-embedding policy before it becomes a standing tax.

5. Caching is a moat, not just a discount. Prompt caching, semantic caching of repeated sub-results, and reusing retrieval across turns don't just lower your bill, they let you price below a competitor who hasn't done the work, at the same gross margin. In a market where two providers sell "the same" agent, the one with the deeper caching and routing discipline can win on price and margin simultaneously. The cost stack is a competitive weapon, not just an accounting exercise.

References

• a16z, LLMflation: LLM inference cost is going down fast
• Gartner, Over 40% of Agentic AI Projects Will Be Canceled by End of 2027