Caching, Memory, and the Quiet Levers of Agent Gross Margin

Table of Contents

• The margin problem nobody puts on the slide
• Why caching is the single biggest lever you control
• The anatomy of a cache hit (and why agents wreck them)
• Memory: the cost you re-pay on every single turn
• Four memory architectures and what they do to COGS
• Putting numbers on it: a worked margin example
• Where caching and memory fight each other
• Insights Most People Overlook
• References

The margin problem nobody puts on the slide

Spend any time reading GaaS pitch decks and you'll notice a pattern. The cost section, if it exists at all, names the model and the per-token price, multiplies by an estimated token count, and calls it a unit cost. Clean. Defensible-looking. Almost always wrong by a factor of two or more.

The reason is that an agent is not a single inference call. It's a loop. It reads context, reasons, calls a tool, reads the result, reasons again, calls another tool, and keeps going until it decides it's done. Every pass through that loop re-sends a growing pile of context to the model, the system prompt, the tool definitions, the conversation so far, the retrieved documents, the intermediate results. By the time a moderately complex task finishes, you may have sent the same 8,000-token system prompt fifteen times. You paid for all fifteen.

This is where gross margin actually gets decided, and it's almost never the part founders are looking at. They're tuning which model to use or arguing about whether to charge per task or per outcome (a genuinely hard question covered in cost-per-completed-task: defining the category's core unit). Meanwhile the thing eating 40% of their inference bill is that they're re-paying for input tokens they already sent a turn ago, because nobody set up caching correctly.

Caching and memory are the quiet levers. They don't show up in a demo. They don't make the agent smarter. They just decide whether you keep half your revenue or a fifth of it.

Why caching is the single biggest lever you control

Prompt caching is the closest thing GaaS has to free money, and it works because of an asymmetry in how transformers process input. The expensive part of a long prompt is the initial forward pass that builds the key-value attention state for every token. If the provider can store that computed state and reuse it when the same prefix shows up again, it can skip the work, and pass most of the savings to you.

The discounts are not marginal. Anthropic's prompt caching documentation prices cache reads at roughly one-tenth the cost of fresh input tokens. OpenAI's automatic prompt caching gives a 50% discount on cached input with no code changes required. Google's context caching on Gemini offers similar economics for large, stable contexts. So the same input token can cost you 100% of list price or 10% of it depending entirely on whether it landed inside a reusable prefix.

Now layer the agent loop on top. A task that loops fifteen times sends its stable prefix, system prompt, tool schemas, few-shot examples, fifteen times. With caching, you pay full freight once and a tenth of that fourteen times. The arithmetic is stark. If your stable prefix is 10,000 tokens and your task loops fifteen times, you've sent 150,000 prefix tokens. Without caching that's 150,000 tokens at full price. With caching it's 10,000 at full price plus 140,000 at a 90% discount, an effective bill of 24,000 token-equivalents. You just cut that portion of your COGS by 84%.

That's not a rounding error. On a support agent or coding agent doing thousands of tasks a day, it's the difference between a business and a hobby. It's also why the framing in the margin trap of "we'll just pass through model costs" is so dangerous, your costs are not a fixed pass-through, they're a function of how well you engineered the prefix.

The anatomy of a cache hit (and why agents wreck them)

Here's the catch, and it's the part most teams discover the expensive way: caching is prefix-based and exact-match. The provider can only reuse the cached state up to the first token that differs. Change one token near the front of your prompt and everything after it is a cache miss, billed at full price.

Agents are remarkably good at accidentally breaking their own caches. A few of the classic self-inflicted wounds:

• Timestamps in the system prompt. Inject "The current time is 2026-06-25 14:33:07" at the top and you've guaranteed a fresh cache on every single call, forever. Move volatile data to the end of the prompt, after the stable cacheable block.
• Reordering tool definitions. If your framework serializes the available tools in a non-deterministic order, the prefix changes between calls and the cache never hits. Pin the order.
• Per-user data near the front. Putting the user's name, account ID, or retrieved context ahead of the shared system instructions fragments your cache across every user. Keep the genuinely shared, high-volume prefix first.
• Cache TTL expiry on idle agents. Provider caches typically live 5 minutes to an hour. An agent that pauses to wait on a slow tool or a human approval can blow past the TTL and re-pay full price on resume. This is one of the hidden taxes discussed in the "idle agent" cost problem and how vendors hide it.

The design principle is simple to state and surprisingly hard to enforce across a codebase: structure every prompt as a long stable prefix followed by a short volatile suffix. Most agent frameworks make it easy to do the opposite. Auditing your actual cache hit rate, providers expose it in the usage response, is the single highest-ROI hour an agent operator can spend. I have seen teams discover their cache hit rate was effectively zero because of one stray timestamp, and recover 35% of their inference bill by moving one line.

Memory: the cost you re-pay on every single turn

If caching is about not re-paying for tokens you've already sent, memory is about deciding which tokens to send in the first place, and it's a thornier problem because there's no free lunch.

An agent that "remembers" prior interactions has to get that memory into the context window somehow, and the context window is metered. Every approach to memory is, underneath, a decision about what to put in front of the model on each turn and what to leave out. That decision is a direct multiplier on your per-turn cost.

The naive approach, keep the entire conversation history in context and grow it forever, is the most expensive thing you can do, and it's the default in a shocking number of frameworks. Cost grows with the square of the interaction length in the worst case, because each new turn re-sends a longer history. A long-running agent (the kind dissected in the economics of long-running agents (hours, not seconds)) that naively accumulates context will see its per-turn cost climb steadily until a single task costs more than ten short ones. The agent gets slower and more expensive precisely as the task gets more important.

So memory is not a feature you add for capability. It's a cost-control discipline. The question is never "should the agent remember", it's "what's the cheapest representation of the past that preserves enough signal to do the job." That reframing is the whole game.

Four memory architectures and what they do to COGS

There's no single right answer, but the four common patterns have very different cost profiles, and most teams pick one by accident rather than on purpose.

Full-history (the expensive default)

Send everything, every turn. Simple, high-fidelity, and ruinous at scale. Cost scales with conversation length, cache helps with the stable front but not the growing tail. Fine for short tasks; a budget grenade for anything long-running. The storage angle here, keeping all that history persisted, has its own economics covered in the economics of agent memory storage at scale.

Rolling-window / truncation

Keep the last N turns, drop the rest. Bounds your per-turn cost cleanly, which finance teams love. The risk is silent capability loss: the agent forgets the constraint it agreed to twenty turns ago and does something dumb. Cheap, but you pay for it in success rate, which connects directly to agent success rate vs. task completion rate, why they differ.

Summarization / compaction

Periodically compress old turns into a dense summary, then carry the summary forward. This is the sweet spot for most production agents. You trade a small recurring summarization cost (one extra LLM call every K turns) for a large, permanent reduction in per-turn context. The art is in the compaction trigger and the summary quality, over-aggressive summarization throws away the detail the agent needed, while lazy summarization defeats the purpose.

Retrieval-based (external memory + RAG)

Store everything outside the context window in a vector or keyword store, and retrieve only the relevant slices on demand. This decouples memory size from per-turn cost entirely, you can have unbounded memory at bounded context cost. The trade is retrieval infrastructure, embedding costs, and the ever-present risk of retrieving the wrong thing. McKinsey's analysis of the economic potential of generative AI repeatedly lands on the same point in practice: the value is real but only captured when the plumbing is engineered for cost, not just capability.

The right answer for most vertical agents is a hybrid: compaction for the working conversation, retrieval for the long tail, and aggressive caching on the stable prefix that all of it shares.

Putting numbers on it: a worked margin example

Let me make this concrete, because the abstractions hide how much money is moving.

Take a customer-support agent priced at $0.50 per resolved ticket (the kind of unit modeled in unit economics teardown: a customer-support agent at scale). A typical ticket runs an 8-turn loop. The stable prefix, system prompt, tool schemas, brand guidelines, policy docs, is 12,000 tokens. The volatile per-turn content averages 2,000 tokens. Say your blended model cost is $3 per million input tokens.

Naive implementation, no caching, full-history memory: Turn 1 sends 14,000 tokens, turn 2 sends 16,000, and so on as history accumulates. Across 8 turns you send roughly 175,000 input tokens. At $3/M that's about $0.53 in input cost alone, before output tokens, before tool calls. You are already underwater on a $0.50 price.

Engineered implementation, caching plus compaction: The 12,000-token prefix is paid once at full price ($0.036) and read from cache seven more times at 10% ($0.025). Compaction keeps the volatile working context flat at roughly 3,000 tokens per turn instead of growing. Total input cost lands near $0.10, output and tools on top, but you're comfortably margin-positive.

Same agent. Same model. Same task. The input-side COGS moved by roughly 5x, and the only things that changed were caching discipline and memory architecture. This is what people mean when they say infrastructure decides GaaS margins, and why benchmarking your own spend, as in benchmarking inference spend across the top 10 agent platforms, tends to reveal that the platforms with the best margins aren't using cheaper models, they're using their tokens better.

Where caching and memory fight each other

The uncomfortable truth is that the two levers can work against each other, and the tension is real.

Caching rewards stability, a long, unchanging prefix that hits the cache turn after turn. Memory compaction, by its nature, changes the context: every time you compact, you rewrite the history, which means the prefix shifts and your cache invalidates. Compact too often and you save on context size but pay full price on every post-compaction call because you keep busting the cache.

The resolution is architectural, not magical. You partition the prompt into zones by volatility. The genuinely static block, system instructions, tool schemas, policies, goes first and stays byte-identical so it caches indefinitely. The compacted memory summary goes in a middle zone, updated only at compaction boundaries, so it caches between compactions. The live turn-by-turn content goes last, where churn is expected and cheap. Get this layering right and the two levers stop fighting and start compounding: stable prefix cached, summary cached between compactions, only the genuinely new tokens paid at full rate.

Most teams never draw this map. They let the framework assemble the prompt however it likes and wonder why their margins are mushy. The operators who win at GaaS economics treat the prompt as a financial document, every token placed deliberately, every zone justified by its volatility and its cost. That discipline, more than model selection, is what shows up at the bottom of the P&L.

Insights Most People Overlook

1. Your cache hit rate is a gross-margin KPI, not an engineering detail. Almost nobody tracks it on the finance side, yet it directly sets your input COGS. A support agent at 90% cache hit rate and one at 20% have completely different P&Ls on identical models. It belongs on the metrics dashboard next to cost-per-task, not buried in an observability tool that only engineers look at.

2. Cheaper models can make your margins worse. Counterintuitive but common: teams downgrade to a cheaper model to cut costs, but the cheaper model is less reliable, retries more, and loops longer, and those extra turns each re-send the full context. You can pay less per token and more per task. The token price is a trap; cost-per-completed-task is the only number that matters.

3. The free tier is uniquely brutal for memory-heavy agents. Because memory cost compounds with usage, a generous free tier on a long-running agent is a way to lose money in proportion to how much people like your product. This is a sharper version of the problem in the free-tier discussion across the cluster, for agents, the marginal cost of an engaged free user is not near zero the way it is in SaaS.

4. Compaction quality is a hidden reliability lever, not just a cost lever. A bad summary doesn't just waste the summarization call, it silently degrades every downstream turn because the agent is now reasoning over a lossy compression of its own history. The cheapest agents and the most reliable agents are often the same agents, because good compaction serves both. Cost and quality are not always a trade-off here; sometimes they're the same investment.

5. Provider cache TTLs quietly tax your slowest, most valuable workflows. The agents that pause for human approval or wait on slow external systems, usually your highest-stakes, highest-value tasks, are exactly the ones most likely to blow the cache TTL and re-pay full price on resume. Your most important work can carry your worst unit economics, and nothing in the dashboard will tell you unless you instrument resume-path cache misses specifically.

References

• Anthropic, Prompt Caching Documentation
• OpenAI, Prompt Caching Guide
• McKinsey, The Economic Potential of Generative AI: The Next Productivity Frontier