The Picks-and-Shovels Map of Agent Infrastructure: Where the Real Money Is Being Made

Table of Contents

• Why "Picks and Shovels" Is the Right Lens
• The Seven Layers of the Agent Infrastructure Map
• Layer 1: The Model and Inference Substrate
• Layer 2: Orchestration and Frameworks
• Layer 3: The Runtime and Durable Execution
• Layer 4: Memory, State, and Retrieval
• Layer 5: Tools, Protocols, and the Integration Fabric
• Layer 6: The Gateway, Identity, and Policy Plane
• Layer 7: Observability, Evals, and Reliability
• Reading the Map: What's Commoditizing vs. What's Open
• How an Operator Should Actually Use This Map
• Insights Most People Overlook
• Frequently Asked Questions
• Conclusion
• References

Why "Picks and Shovels" Is the Right Lens

The picks-and-shovels framing gets overused in tech, so it's worth being precise about why it fits here. In a normal software wave, the application and the infrastructure are loosely coupled, you can build a great SaaS product on boring, well-understood plumbing. Agentic systems break that comfort. An agent that runs for forty minutes, calls fifteen tools, spends real money per task, and occasionally hallucinates a destructive action puts enormous, unusual demands on its infrastructure. The plumbing isn't boring anymore; it's the difference between a margin-positive product and one that quietly loses money on every run.

That's what makes the infrastructure layer investable and worth understanding even if you never sell it. Andreessen Horowitz has argued for a while that the value in AI may concentrate in the infrastructure and tooling layers rather than the application surface, precisely because the hard, repeatable problems live there. For a GaaS operator, the map matters for a more practical reason: the cost stack, the reliability ceiling, and the security posture of your product are all decided by infrastructure choices you make early and rarely revisit. Get them wrong and you find out at scale, when changing them is expensive.

So this is both a market map (who's selling shovels) and an architecture map (what an agent product is actually made of). The two are the same diagram viewed from different chairs.

The Seven Layers of the Agent Infrastructure Map

Most attempts to draw the agent stack either oversimplify it into "model + framework" or explode it into a logo soup of forty vendors. Neither helps an operator decide anything. The useful resolution is seven layers, each defined by a distinct job and a distinct failure mode:

1. Model and inference substrate, where tokens are generated.
2. Orchestration and frameworks, how reasoning and tool calls are sequenced.
3. Runtime and durable execution, how a long-running agent survives crashes and restarts.
4. Memory, state, and retrieval, what the agent knows across and within tasks.
5. Tools, protocols, and integration fabric, how the agent acts on the world.
6. Gateway, identity, and policy plane, who's allowed to do what, and at what rate.
7. Observability, evals, and reliability, how you know it's working and catch it when it isn't.

The rest of this piece walks each layer: what it does, why it's hard, and where the genuine opportunity sits.

Layer 1: The Model and Inference Substrate

This is the most visible layer and, counterintuitively, the one an operator should think about least often once chosen, but get badly wrong most expensively. The substrate is the frontier and open-weight model providers plus the inference companies serving them: managed APIs, GPU clouds, and specialized fast-inference shops.

The picks-and-shovels reality here is brutal and well known: the people selling compute are doing fine regardless of which agent app wins. The constraint underneath everything is GPU supply, which is why the economics of large-scale model serving remain dominated by hardware availability and utilization, not software cleverness. For a GaaS business, the substrate decision is really three sub-decisions that get conflated: which model family, served by whom, and accessed through what routing logic. The third is where margin hides, a model-routing layer that sends easy steps to a cheap model and only escalates hard ones to a frontier model can cut inference cost by half without the user noticing. That single technique is a cluster topic in its own right, and it's where naive agent products bleed money.

The mistake operators make is treating the substrate as a permanent commitment. It isn't. The interface to a model is narrow enough that the substrate should be the most swappable part of your stack, provided you didn't hardwire one provider's quirks into your orchestration. Designing a multi-model agent that avoids vendor lock-in is a deliberate architectural choice, not a default you get for free.

Layer 2: Orchestration and Frameworks

Orchestration is where the agent's "thinking loop" lives: the code that decides what to do next, calls a tool, reads the result, and loops. This is the layer everyone argues about because it's the one developers touch directly. The framework landscape, LangChain, LlamaIndex, and a growing field of challengers, competes here, alongside the newer crop of orchestration-first frameworks built specifically for multi-step agents rather than retrofitted from RAG pipelines.

Here's the uncomfortable truth this layer keeps proving: frameworks are sticky in adoption but thin in defensibility. The core agent loop is a few hundred lines of code. What frameworks actually sell is the integration surface around it, the connectors, the memory adapters, the observability hooks, not the loop itself. That's why the framework wars feel perpetually unsettled. The orchestration pattern you choose (a single agent with tools, a supervisor coordinating sub-agents, or an event-driven async mesh) matters far more to your reliability than which library you import. A supervisor-agent architecture and a flat tool-calling agent fail in completely different ways, and choosing between them is an engineering decision the framework mostly doesn't make for you.

For an operator, the practical guidance is to treat the framework as scaffolding, not foundation. Pick one that doesn't fight you, but design your orchestration logic so you could swap the framework without rewriting your business rules. The teams that regret their framework choice almost always regret coupling their domain logic to it, not the choice itself.

Layer 3: The Runtime and Durable Execution

This is the layer most newcomers don't know exists until it hurts them. A traditional web request lives for milliseconds. An agent task can live for minutes or hours, spanning dozens of model calls and tool invocations, any of which can fail. What happens when your server restarts mid-task? Without a durable execution layer, the agent simply dies and the work is lost, along with whatever the customer paid for.

Durable execution engines solve this by checkpointing the agent's progress so it can resume exactly where it left off, even across crashes and deploys. This is the conceptual home of the emerging "agent runtime" as a distinct infrastructure category, a managed environment that handles long-running execution, state persistence, retries, and human-in-the-loop pauses as first-class concerns rather than things you bolt on. The pattern borrows heavily from durable workflow engines that predate the agent wave, which is why some of the strongest entrants here are workflow-orchestration companies repositioning toward agents.

Why does this matter for GaaS specifically? Because per-outcome pricing only works if outcomes reliably complete. If 8% of your long-running tasks silently die on infrastructure failures, your unit economics and your reputation both degrade. Durable execution turns "the agent crashed" from a customer-facing failure into an invisible retry. It is, quietly, one of the highest-leverage shovels on the whole map.

Layer 4: Memory, State, and Retrieval

Agents need three different things that get lumped together as "memory," and conflating them is a common architectural error. There's short-term state (the working context of the current task), long-term memory (what the agent remembers across sessions), and retrieval (pulling in external knowledge on demand). Each is a separate system with separate tradeoffs.

Short-term state is really a context-window management problem, deciding what to keep, summarize, or evict as the agent's working memory fills up. This is the context-window economy, and it's where token budgets quietly explode. Long-term memory is an architecture choice with a privacy tradeoff baked in: the more an agent persistently remembers about a user, the more valuable and the more dangerous that store becomes. Retrieval, meanwhile, has moved well beyond basic RAG, agents increasingly need structured retrieval, multi-hop lookups, and the ability to decide when to retrieve, not just what.

The role of vector databases in this layer is genuinely contested right now. For a while, a vector DB was assumed mandatory. That assumption is softening as memory systems mature and as some retrieval moves to structured stores or hybrid approaches. The honest answer is that vector search remains useful for certain retrieval patterns but is no longer the automatic center of the memory stack it was treated as in 2023. An operator should resist installing one reflexively and instead ask what kind of memory the product actually needs.

Layer 5: Tools, Protocols, and the Integration Fabric

An agent that can't act on the world is a chatbot. Tools are how agents do things, call an API, run code, browse a page, operate a computer. This layer is where the most consequential standardization fight of the entire stack is happening.

The Model Context Protocol (MCP) has emerged as a leading attempt to standardize how agents discover and call tools, the way HTTP standardized how clients talk to servers. Anthropic's introduction of MCP as an open standard for connecting models to tools and data reframed integration from a per-vendor chore into something closer to a shared protocol. Alongside it, agent-to-agent (A2A) protocols are trying to standardize how agents talk to each other, not just to tools. Whoever's standards win this layer captures enormous leverage, which is exactly why protocol fragmentation is currently the norm and consolidation is the predicted endgame.

Underneath the protocol debate sits unglamorous, essential work: making tool calls reliable. Models still call tools with malformed arguments, hallucinate parameters, or misread results. The infrastructure that validates, retries, and sandboxes tool calls, and the API-to-agent adaptation layer that wraps legacy SaaS endpoints into agent-callable tools, is where a lot of real engineering happens. Browser automation and computer-use integration extend this further, letting agents operate interfaces that were never designed for machines. These are picks-and-shovels businesses in the purest sense: nobody markets them, every serious agent needs them.

Layer 6: The Gateway, Identity, and Policy Plane

The moment an agent can spend money, send email, or touch production systems, you need a control plane in front of it. This layer answers three questions every autonomous system raises: who is this agent acting as, what is it allowed to do, and how fast can it do it?

Identity and auth for agents is genuinely unsolved in a clean way. An agent acting on a user's behalf needs delegated credentials that are scoped, auditable, and revocable, and the existing OAuth-era machinery wasn't designed for a non-human actor making thousands of calls. Agent gateways sit at this chokepoint, handling routing, rate-limiting, and policy enforcement: blocking disallowed actions, capping spend, and logging everything for audit. For enterprise deployments, this hardens into the "agent mesh" concept, a managed fabric where many agents operate under centralized identity and policy.

For a GaaS operator, this layer is where trust is won or lost. Enterprise buyers will ask, early and pointedly, how you prevent an agent from doing something catastrophic. A credible answer, scoped credentials, a policy gateway, hard spend caps, full audit logs, is increasingly table stakes, not a differentiator. The infrastructure security posture of autonomous systems is becoming a procurement gate, and the vendors selling that control plane are selling some of the most defensible shovels on the map.

Layer 7: Observability, Evals, and Reliability

You cannot operate what you cannot see, and agents are unusually hard to see into. A single task produces a sprawling trace: prompts, model responses, tool calls, retries, branching decisions. When something goes wrong, and with autonomous systems, something always eventually goes wrong, you need to reconstruct exactly what the agent did and why.

The observability stack for agents has matured fast, borrowing telemetry standards from the broader software world while adding agent-specific concepts: trace trees, token accounting per step, and eval harnesses that score agent behavior against expected outcomes. This is also where reliability infrastructure lives, retries, fallbacks, circuit breakers, the same patterns that made distributed systems dependable, now applied to a substrate that's probabilistic rather than deterministic. The wrinkle is that agents fail silently and plausibly: they don't crash, they produce a confident wrong answer. That makes evals and simulation environments more important here than in conventional software, because the only way to catch quality regression is to test behavior, not just uptime.

For operators selling on a per-outcome basis, observability isn't optional tooling, it's how you prove the outcome happened, defend against disputes, and find the 5% of failure modes eating your margin. It closes the loop on the whole map.

Reading the Map: What's Commoditizing vs. What's Open

A map is only useful if it tells you where to point. Here's how the seven layers sort by maturity, as of now:

Commoditizing fast (don't build, just buy): the inference substrate and the basic orchestration loop. These are racing toward zero differentiation. Build your business on top of them, not in them.

Contested and consolidating: frameworks, protocols (MCP, A2A), and vector/retrieval. There's real value here but the winners aren't settled, so avoid deep coupling. Bet on standards, hedge on specific vendors.

Open and underbuilt (where opportunity concentrates): durable execution and runtimes, the gateway/identity/policy plane, and agent-specific observability and evals. These solve problems that only show up at production scale, which is exactly why they're underserved, most teams haven't hit the pain yet. The shovels with the most pricing power are here.

This sorting will shift; an annual stack report is the right cadence for tracking it. But the directional logic, value migrates toward the hard, production-only problems, has held through every prior infrastructure wave.

How an Operator Should Actually Use This Map

If you run or are building a GaaS product, the practical move is to audit your stack layer by layer and ask one question at each: am I exposed here? Most operators discover they've over-invested in the visible layers (model choice, framework) and under-invested in the invisible ones (durable execution, gateway, observability) that actually determine whether the product survives contact with real traffic.

A reasonable sequence: lock down identity and spend caps before you scale anything, because that's the layer that can hurt customers. Add durable execution before your tasks get long, because retrofitting it is painful. Instrument observability from day one, because you can't backfill traces you never captured. Treat the model and framework as swappable, because they will change under you whether you plan for it or not. The infrastructure cost stack, decomposed honestly, almost always reveals that the cheap-to-add reliability layers prevent the expensive-to-fix failures.

The deeper point connects back to the whole GaaS cluster: agent economics, agent reliability, and agent security aren't separate topics from infrastructure, they're consequences of it. The picks-and-shovels map isn't a vendor shopping list. It's the diagram of where your product's margins, uptime, and trust are actually decided.

Insights Most People Overlook

The most defensible shovels are the least demoable. Durable execution, policy gateways, and eval harnesses don't make good launch demos, they prevent invisible disasters rather than enabling visible magic. That's precisely why they're underbuilt and why they hold pricing power. The infrastructure that wins long-term tends to be the infrastructure that's boring to show and painful to live without.

Framework lock-in is a myth; orchestration-logic lock-in is the real trap. Teams obsess over which framework to pick, then couple their domain logic so tightly to it that switching becomes a rewrite. The framework is replaceable in a weekend if you kept your business rules separate. The thing you actually can't escape is the pattern you chose, supervisor vs. flat vs. event-driven, because that shapes your whole failure surface.

Vector databases became infrastructure by assumption, not by necessity. A surprising amount of agent memory doesn't need similarity search at all, it needs structured recall, recency, or simple key-value state. The reflexive "add a vector DB" move installs cost and complexity that many products never recoup. Ask what the memory is for before reaching for embeddings.

Per-outcome pricing is a bet on your durable-execution layer. Operators love outcome-based pricing as a sales story, then under-resource the infrastructure that makes outcomes reliably complete. If a meaningful slice of your long-running tasks die on transient failures, outcome pricing turns from a margin advantage into a margin trap. The pricing model and the runtime layer are the same decision wearing two hats.

The gateway layer is becoming a procurement gate, not a feature. Enterprise buyers increasingly won't sign without scoped credentials, spend caps, and audit logs for autonomous actions. Treating agent identity and policy as a "later" problem doesn't just add risk, it closes deals you'll never know you lost, because the buyer disqualified you in the security review before you got to the demo.

Frequently Asked Questions

Is the agent infrastructure stack the same as the traditional ML/MLOps stack? No, and assuming so causes real mistakes. MLOps centers on training, deploying, and monitoring models. Agent infrastructure centers on runtime behavior, orchestrating tool calls, managing long-running state, enforcing policy on autonomous actions. They overlap at observability but diverge sharply everywhere else. An agent product needs durable execution and a policy gateway that classical MLOps never had to provide.

Do I need a dedicated agent runtime, or can I run agents on normal application servers? You can start on normal servers, but you'll hit a wall once tasks run long enough to outlive a deploy or a crash. The signal that you need a dedicated runtime is when losing in-flight tasks becomes a customer-facing problem. Below that threshold, normal infrastructure plus careful state-saving is fine; above it, durable execution stops being optional.

Where does MCP fit, and should I adopt it now? MCP sits at the tool-integration layer, standardizing how agents discover and call tools. Adopting it now is reasonable if you have many integrations to maintain, because it reduces per-tool custom work. The risk is protocol fragmentation, competing standards may coexist for a while, so design your tool layer so the protocol is swappable rather than betting everything on one winning immediately.

How much of this stack should I build versus buy? Buy the commoditizing layers (inference, basic orchestration) without hesitation. Be deliberate about the open layers (runtime, gateway, observability), buying gets you to production faster, building gives you control if those layers are core to your differentiation. The wrong answer is building the substrate yourself, which almost never pays off, or hand-rolling observability, which you'll regret the first time you debug a production failure blind.

What's the single most underestimated cost in agent infrastructure? Token spend driven by unmanaged context. Agents accumulate context as they run, and naive implementations resend ever-growing histories on every step, multiplying cost. Context-window management and caching aren't optimizations you add later, they're the difference between a profitable per-task price and a losing one. Most operators discover this only after their first surprising invoice.

How do agent security concerns differ from normal application security? A normal app does what its code says. An agent decides what to do at runtime, which means it can be manipulated into actions its authors never intended, through prompt injection, poisoned tool results, or compromised data. That shifts security from "secure the code" to "constrain the autonomy": scoped credentials, sandboxing, spend caps, and human checkpoints for high-stakes actions. The gateway and policy layer exists precisely because traditional app security assumptions don't hold for autonomous systems.

Conclusion

The agent gold rush will mint a few application winners and a lot of forgotten ones. But underneath all of them runs an infrastructure layer that gets paid regardless, the picks and shovels of autonomy. That layer resolves into seven jobs: generating tokens, orchestrating reasoning, executing durably, managing memory, integrating tools, enforcing policy, and observing behavior. The visible layers are commoditizing; the value is migrating toward the hard, production-only problems most teams haven't hit yet, durable execution, the identity-and-policy gateway, and agent-specific observability.

For a GaaS operator, the takeaway isn't a vendor list. It's a way of seeing: your product's economics, reliability, and security aren't downstream of infrastructure choices, they are those choices, made early and rarely revisited. Map your own stack against these seven layers, find where you're exposed, and reinforce the unglamorous layers before scale forces the question. The operators who treat infrastructure as the product's spine rather than its plumbing are the ones who'll still be standing when the rush thins out. The rest of this cluster, agent economics, reliability engineering, security, and orchestration patterns, is really just a closer look at individual shovels on this same map.

References

• Andreessen Horowitz, Who Owns the Generative AI Platform?
• Anthropic, Introducing the Model Context Protocol
• SemiAnalysis, Coverage of AI compute economics and GPU supply