Inference Optimization for Agent Workloads: Where the Real Money and Milliseconds Hide

Table of Contents

• Why Agent Inference Is a Different Animal
• The Two Phases Every Optimization Targets
• Prefix Caching: The Single Highest-Leverage Win
• Batching for Bursty, Bursty Agent Traffic
• Speculative Decoding and Why Agents Love It
• Quantization Without Wrecking Tool-Call Accuracy
• The Context-Length Tax and How to Stop Paying It
• Buy vs. Build: Managed Endpoints vs. Self-Hosted Serving
• Measuring the Right Things
• Insights Most People Overlook
• References

Why Agent Inference Is a Different Animal

If you came to agents from chatbot or RAG work, your mental model of inference is probably wrong in ways that quietly cost you money.

A conversational app issues one model call and then waits -- often for many seconds -- while a human reads, thinks, and types. Latency tolerance is generous. Token volume is modest. The traffic pattern is roughly Poisson and easy to batch.

An agent does none of that. A single "task" in an Agentic AI-as-a-Service product -- triage a support ticket, reconcile an invoice, draft and send a sequence of emails -- typically spins through a loop: read context, call the model, parse a tool call, run the tool, feed the result back, call the model again. Five to fifteen model calls per task is common; complex research or coding agents run into the dozens. Critically, those calls are chained. Call N+1 cannot start until call N returns. So latency compounds. A 1.5-second time-to-first-token feels fine in a chat window and feels like molasses when you stack it twelve times in a loop the user is staring at.

The second difference is the one that hits the invoice: agents re-send enormous amounts of identical context. The system prompt, the tool definitions, the task instructions, the accumulated scratchpad -- all of it gets shipped to the model on every single step. In a ten-step loop with a 6,000-token preamble, you've paid to process that preamble ten times. For GaaS operators pricing per task or per outcome, that duplication is the difference between a healthy margin and a loss. This is the same pressure that drives the broader GaaS infrastructure cost stack; inference is usually the single largest line item in it.

So "inference optimization for agent workloads" isn't generic LLM serving with a new label. It's a specific discipline shaped by chained calls, heavy context reuse, bursty concurrency, and unforgiving unit economics.

The Two Phases Every Optimization Targets

Almost every technique below is really attacking one of two phases of how a transformer generates text. Understanding the split makes the rest obvious.

Prefill (the prompt-processing phase). When you send a prompt, the model processes all input tokens in parallel to build up its internal key-value (KV) cache. This is compute-bound -- it saturates the GPU's math units. The metric it controls is time-to-first-token (TTFT).

Decode (the generation phase). Then the model emits output tokens one at a time, each pass reading the entire KV cache from memory. This is memory-bandwidth-bound, not compute-bound. The metric it controls is inter-token latency and overall throughput.

Why this matters for agents: agent loops are prefill-heavy relative to chatbots. They re-send big prompts (lots of prefill) and frequently emit short, structured outputs -- a JSON tool call, a single decision token (relatively little decode). That ratio is unusual, and it changes which optimizations pay off. Techniques that speed up prefill or let you skip it entirely tend to deliver the biggest wins for agents, which is exactly why prefix caching tops the list.

Prefix Caching: The Single Highest-Leverage Win

If you do one thing, do this.

Prefix caching (sometimes "prompt caching" or "context caching") stores the KV cache computed for a shared prompt prefix so it can be reused instead of recomputed. Because every step of an agent loop shares the same leading context -- system prompt, tools, task setup -- the prefill for that chunk should be computed once and reused on every subsequent call.

The savings are not marginal. The major providers have productized this with steep discounts on cached input. Anthropic's prompt caching documentation charges cache reads at a fraction of normal input-token price, and OpenAI's prompt caching guide applies automatic discounts on repeated prefixes. On self-hosted stacks, vLLM's automatic prefix caching does the same thing without per-call billing -- you just reclaim the GPU cycles.

Two practical rules separate teams who get 5% from teams who get 50%:

Order your context for cache hits. Caches match on prefixes, so anything that changes invalidates everything after it. Put the stable material first -- system prompt, tool definitions, few-shot examples -- and the volatile material (the current step's tool result, the user's latest message) last. A surprising number of agent frameworks shuffle tool definitions or inject timestamps near the top of the prompt and silently destroy the cache on every call.

Mind the cache lifetime. Provider caches expire fast (often five minutes of inactivity). An agent that pauses for a slow tool call or a human-in-the-loop checkpoint can blow past the TTL and re-pay full prefill. For long-running agents this interacts directly with the orchestration challenge of long-running execution -- sometimes you keep a cheap keep-alive call alive, sometimes you accept the re-prefill.

Prefix caching is also why naive caching strategies that cut agent costs and inference optimization overlap so much: a response cache skips the call entirely; a prefix cache makes the calls you do make far cheaper.

Batching for Bursty, Bursty Agent Traffic

Static batching -- wait for N requests, run them together -- was built for offline jobs. It's actively bad for agents, where requests arrive unpredictably and one slow generation holds the whole batch hostage.

Continuous batching (also called in-flight batching) is the standard answer. Instead of waiting for a fixed batch, the server adds and evicts requests at the token level: as soon as one sequence finishes, a new one takes its slot. This keeps the GPU busy and is the default in serious serving stacks. The original throughput gains documented in the vLLM PagedAttention work came largely from combining continuous batching with smarter KV-cache memory management.

For agent workloads specifically, watch two things. First, concurrency is spiky -- a GaaS product might be idle, then fire 200 agents at once when a batch of tickets lands. Provision and autoscale for the bursts, not the average, or your p99 TTFT explodes exactly when it matters. Second, prefill can starve decode. A flood of new agent steps (all prefill) can stall the token generation of in-flight requests. Modern stacks mitigate this with chunked prefill, which interleaves prefill work with decode so long prompts don't freeze everyone else. If your serving layer doesn't expose that knob, large agent prompts will hurt tail latency for the whole fleet.

Speculative Decoding and Why Agents Love It

Speculative decoding uses a small, fast "draft" model to guess several tokens ahead, then has the large target model verify those guesses in a single forward pass. When the draft is right, you get multiple tokens for the price of one decode step. When it's wrong, you fall back -- no accuracy loss, just less speedup.

Here's the under-appreciated part for agents: their output is unusually predictable, which makes speculation hit rates high. A lot of agent output is structured and low-entropy -- JSON keys you've seen a thousand times ("tool_name":, "arguments":), boilerplate field names, repeated formatting. A cheap draft model nails that, so acceptance rates on tool-call-heavy output run well above what you'd see on freeform prose. The DeepMind paper introducing the technique, Accelerating Large Language Model Decoding with Speculative Sampling, reported 2-2.5x speedups on general text; structured agent output often does better.

The catch: speculative decoding helps the decode phase, and agents are prefill-heavy. So it's a real win for agents that generate long structured outputs (code, large JSON payloads, multi-field forms) and a smaller one for agents that emit a single decision token per step. Match the technique to your output profile rather than turning it on reflexively.

Quantization Without Wrecking Tool-Call Accuracy

Quantization shrinks model weights (and sometimes activations and the KV cache) to lower precision -- FP8, INT8, INT4 -- cutting memory footprint and bandwidth pressure. Less memory means more requests fit per GPU and decode runs faster. For cost-sensitive GaaS operators it's tempting to crank it.

Be careful, because agents are sensitive in a way chatbots aren't. A chatbot that's 1% less eloquent is fine. An agent that, 1% more often, emits malformed JSON or picks the wrong tool can fail an entire task -- and tool-call correctness is exactly the kind of structured behavior that degrades first under aggressive quantization. The failure isn't "slightly worse prose," it's a broken loop, a retry, or a wrong action taken in the world.

Practical guidance: FP8 and well-implemented INT8 are generally safe for agent workloads and worth taking. Below that -- 4-bit and aggressive activation quantization -- validate against your tool-calling eval set, not a generic benchmark. The thing that breaks is rarely perplexity; it's structured-output fidelity, which standard benchmarks barely measure. This connects to the broader question of tool-calling reliability at the infrastructure layer: quantization is one of several places where an infra decision silently erodes reliability unless you're measuring the right thing.

The Context-Length Tax and How to Stop Paying It

The single biggest controllable cost in an agent loop is how much context you carry forward. Attention cost grows with sequence length, the KV cache grows linearly, and -- as covered above -- you re-pay for context on every step it survives. An agent that naively appends every tool result to a growing transcript gets slower and more expensive on every iteration, a compounding tax that's invisible until you read the bill.

The optimizations here are partly architectural, but they're inseparable from inference performance:

• Trim aggressively between steps. Most tool outputs don't need to live in context forever. Summarize or drop stale results. The context-window economy is its own discipline, but the inference payoff is immediate: shorter prompts mean cheaper prefill and a smaller KV cache.
• Externalize memory. Push durable state into a memory system or a retrieval layer and pull only what the current step needs, rather than keeping everything resident in the prompt.
• Watch the KV cache, not just token count. On self-hosted stacks the KV cache is often the real capacity ceiling -- it's what limits how many agents run concurrently per GPU. Long contexts eat it fast. Techniques that compress or page the KV cache (PagedAttention, KV-cache quantization) directly raise the number of agents you can serve on the same hardware.

The runtime version of this discipline -- enforcing token budgets per step, evicting context dynamically -- is increasingly handled by the agent runtime itself rather than left to application code.

Buy vs. Build: Managed Endpoints vs. Self-Hosted Serving

A real strategic fork, and the right answer changes as you scale.

Managed endpoints (the model providers' APIs, or serverless inference platforms) give you prompt caching, batching, and quantization for free, maintained by people who do nothing else. You pay per token, you get zero ops burden, and for most teams under serious volume this is correct. The downside is per-token economics that look worse the more you scale, plus less control over tail latency and cache TTLs.

Self-hosted serving (vLLM, TensorRT-LLM, SGLang on your own or rented GPUs) flips the trade. You own continuous batching, you tune prefix-cache retention to your agents' actual loop timing, you pick your quantization, and at high, steady volume the unit economics can be dramatically better. The cost is real operational weight -- GPU supply, autoscaling, the GPU-supply constraint on scaling agents -- and the self-hosted vs. managed infrastructure decision deserves its own analysis.

The pattern I'd bet on: start managed, instrument relentlessly, and move your highest-volume, most-predictable agent paths to self-hosted serving once the token bill clearly exceeds the loaded cost of running your own. Plenty of teams self-host their cheap high-volume worker model while routing hard reasoning steps to a managed frontier API -- which is really a model-routing decision wearing an infrastructure hat.

Measuring the Right Things

You can't optimize agent inference with chatbot metrics. The ones that actually matter:

• TTFT (time-to-first-token), measured per step and compounded across the loop. The per-step number lies; the task-level total is what the user feels.
• Cache hit rate. If your prefix cache hit rate isn't high, your context ordering is broken. This is the first number to check and the cheapest to fix.
• Cost per task, not cost per token. Token price is an input; task cost is the unit your GaaS pricing has to beat. Two stacks with identical token prices can differ 3x on cost per task purely on caching and context discipline.
• Tool-call success rate under your serving config. The metric that tells you whether quantization or batching changes quietly broke reliability. Track it as a first-class part of your observability stack.

Optimize against task-level cost and latency, and the right techniques select themselves. Optimize against per-token price alone, and you'll make locally smart, globally expensive decisions.

Insights Most People Overlook

Cache-aware routing beats cheapest-endpoint routing. Most routing logic picks the cheapest model that can do the job. But if request A primed a prefix cache on GPU 3, sending the agent's next step to GPU 3 -- even a nominally "more expensive" node -- can be cheaper overall because it skips re-prefill. Cache locality is a routing input almost nobody uses, and for sticky agent loops it can dominate the math.

Prefill, not decode, is the agent bottleneck -- so half the popular advice is aimed at the wrong phase. Speculative decoding, decode-side quantization, throughput-per-second benchmarks: all decode-phase wins, all the ones that get written up. Agents are prefill-heavy. The techniques that actually move their numbers -- prefix caching, chunked prefill, context trimming -- are the boring prefill ones. If you're benchmarking tokens-per-second to evaluate an agent stack, you're measuring the phase that matters least.

Your framework may be silently busting your cache. Some popular agent frameworks inject dynamic content -- timestamps, randomized tool ordering, request IDs -- near the top of the prompt. Each injection invalidates the entire downstream prefix cache, turning a should-be-90%-discount into full price on every call. This is a five-minute fix that teams overlook for months because nothing breaks -- the bill is just quietly 3x too high.

Latency variance hurts agents more than latency. A chatbot averages out a slow response. An agent's chained loop accumulates tail latency -- a p99 spike on step 4 stalls steps 5 through 12. Optimizing mean TTFT while ignoring p99 makes demos look great and production feel broken. For agents, tightening the tail is often worth more than lowering the average.

Cheaper inference can make agents dumber in non-obvious ways. Aggressive quantization and overzealous context trimming both save money and both degrade exactly the structured, long-range behaviors agents depend on -- correct tool calls, faithful multi-step reasoning. The savings show up instantly on the invoice; the reliability cost shows up later as failed tasks and retries that erase the savings. Always pair an inference cost cut with a tool-call eval, or you're optimizing a number that isn't the one that pays you.

References

• Prompt caching, Anthropic Documentation
• Prompt caching, OpenAI Platform Documentation
• Automatic Prefix Caching, vLLM Documentation
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention, vLLM Blog
• Accelerating Large Language Model Decoding with Speculative Sampling, Chen et al., arXiv