Margin Expansion via Model Routing: Use the Cheap Model When You Can

Table of Contents

• Why Margin Is the Whole Game in GaaS
• What Model Routing Actually Means
• The Three Routing Strategies That Matter
• Where the Margin Actually Comes From
• The Quality Floor Problem
• How to Build a Router That Pays for Itself
• Routing and Your Pricing Model Are Coupled
• Insights Most People Overlook
• References

Why Margin Is the Whole Game in GaaS

Software companies got spoiled. For two decades, the marginal cost of serving one more SaaS customer rounded to zero, and gross margins of 80-90% were the price of admission to a venture round. Agentic AI-as-a-Service broke that spreadsheet. Every task an agent runs burns tokens, and tokens cost real money that scales linearly with usage. A GaaS vendor that charges $2 per resolved support ticket but spends $1.40 on model calls to resolve it is running a 30% gross margin business dressed up as software.

That gap is why margin discipline has become the defining operational skill of this category. You can raise prices, but outcome-based pricing and metered models put a ceiling on how aggressive you can be before buyers revolt. You can renegotiate with your model provider, but you don't have that leverage until you're spending seven figures a month. The lever you control on day one, at any scale, is what you spend per task. And the largest line item in that spend is almost always the model you chose to do the work.

This is where routing enters. The premise is almost embarrassingly simple: most of what an agent does is not hard. Classifying an intent, extracting a date, deciding whether a tool call succeeded, summarizing a short thread, these are tasks a model costing a fraction of a cent per call handles fine. You don't need a frontier reasoning model to determine that "where's my refund" is a billing question. Yet a naive agent architecture sends every single step to the most capable, most expensive model available, because that was the easiest thing to build first. Routing is the practice of fixing that.

What Model Routing Actually Means

Model routing is the decision layer that picks which model handles a given request, based on the difficulty, sensitivity, or value of that request. Instead of a static "always use GPT-class-flagship" wiring, the agent runtime evaluates each step and dispatches it to the cheapest model that can do the job acceptably.

The economics that make this worth doing are stark. Within a single provider's lineup, the price gap between the small model and the flagship is often 15-30x per token. Across the broader market, the spread between a budget open-weight model served on commodity infrastructure and a top-tier proprietary model can exceed 50x. When 75% of your agent's steps could run on the cheap tier without anyone noticing, the blended cost per task collapses even though your hardest steps still cost full freight. Provider pricing pages, for instance, Anthropic's published model pricing and OpenAI's API pricing, make the per-model spread concrete, and it is the entire reason routing exists as a discipline.

The important mental shift is that routing operates at the level of the individual step, not the whole job. A single customer task, say, "process this insurance claim", might decompose into a dozen model calls. Routing asks the question fresh at each one. The intent classification goes cheap. The document extraction goes mid-tier. The one ambiguous coverage-eligibility judgment that actually requires reasoning goes to the flagship. Average them out and you've paid frontier prices for one step out of twelve instead of all twelve.

The Three Routing Strategies That Matter

Most production routers fall into one of three patterns, and they're not mutually exclusive.

Static Tiered Routing

The simplest approach assigns model tiers to task types up front. Your engineers map known step types to model tiers based on offline evaluation: classification and extraction always go to the small model, generation goes mid-tier, final-answer synthesis goes to the flagship. There's no runtime decision-making; the routing is baked into the workflow graph. This is crude but shockingly effective, and it's where every team should start. You capture 80% of the available margin with near-zero risk of a misrouting surprise, because a human decided each route deliberately.

Cascade (Escalation) Routing

A cascade tries the cheap model first and escalates only when the cheap model's output fails a confidence or validation check. The model attempts the task; a verifier, sometimes the cheap model's own self-reported confidence, sometimes a separate check, sometimes a deterministic validator like "did the JSON parse", decides whether to accept the answer or retry with a stronger model. Cascades are elegant because they spend expensive tokens only on the cases that genuinely need them. The trap is that you pay for the cheap attempt on every escalated task, so a poorly tuned cascade with a high escalation rate can cost more than just using the big model directly. The research on FrugalGPT and LLM cascades from Stanford showed cost reductions up to 98% on some workloads, but those numbers assume a well-calibrated escalation threshold, which is the hard part.

Learned Routing

The most sophisticated pattern uses a small classifier, often a fine-tuned lightweight model or even a logistic regression over prompt features, to predict, before any expensive call, whether a request needs the big model. The router learns from historical data which prompts the cheap model handles well and which it flubs. Done right, this captures the cascade's selectivity without paying for failed cheap attempts. Done wrong, it's an opaque box that mis-routes high-value tasks and you can't easily explain why. Learned routers earn their complexity only at scale, where the volume justifies the eval infrastructure to keep them honest.

Where the Margin Actually Comes From

It's worth being precise about the source of the savings, because teams routinely overestimate the win and then get surprised by the bill.

The margin comes from three compounding effects. First, the obvious one: cheaper per-token pricing on routed steps. Second, and underappreciated, smaller models are usually faster, which means lower latency and, if you're self-hosting, better GPU utilization, so you serve more throughput per dollar of fixed infrastructure. Third, smaller models often produce shorter, more constrained outputs for simple tasks, so you're not just paying less per token, you're generating fewer tokens.

But there's a subtraction nobody puts on the slide. Routing adds overhead. The router itself costs something to run, a learned router is another model call. Cascades pay for failed attempts. Verifiers and confidence checks aren't free. And the engineering time to build and maintain the eval harness that keeps routing safe is a real, ongoing cost that doesn't show up in your cloud bill. A team that "saves 60% on inference" but spends two senior engineers' time keeping the router calibrated has not saved as much as the dashboard claims.

The honest framing: routing reliably delivers a large net margin gain on high-volume, repetitive agent workloads, support, document processing, data enrichment, the boring profitable stuff. On low-volume, high-variance, high-stakes workloads, the overhead can eat the savings, and you're better off just running the good model everywhere and pricing for it. McKinsey's analysis of the economic potential of generative AI makes the broader point that value concentrates in a handful of high-volume use cases, and those are exactly the ones where routing pays.

The Quality Floor Problem

Here is where most routing projects quietly go wrong. The cheap model is cheap because it's worse. On the 75% of steps where it's good enough, nobody notices. On the marginal steps, the ones near the boundary of its competence, it produces answers that are plausible but subtly wrong. And subtle wrongness is the worst failure mode in a GaaS product, because it doesn't throw an error. It silently degrades the outcome the customer is paying you to deliver.

A support agent routed too aggressively starts giving technically-accurate-but-unhelpful answers that frustrate users without tripping any alarm. A data-extraction agent starts missing edge-case fields that only show up in 3% of documents, the exact 3% that matter most. The cost dashboard looks great. The quality has regressed, and you find out from a churn spike or a refund request three weeks later, by which point you've trained your customers to distrust the agent.

This is the central tension of margin-via-routing: every dollar of margin you extract by going cheaper is a dollar of quality risk you're taking on, and the relationship is non-linear. The first chunk of routing is nearly free quality-wise. Push too far and quality falls off a cliff. The job is to find the knee of that curve and stop, and you cannot find it without continuous evaluation against a ground-truth set. Routing without an eval harness isn't optimization; it's gambling with your retention numbers.

How to Build a Router That Pays for Itself

A practical sequence, in the order that actually works:

Start with measurement, not routing. Before you route anything, instrument cost and a quality metric per step type. You cannot optimize what you can't see, and you need a quality baseline to detect regressions. If you skip this, you'll never know whether routing helped or just made you cheaper and worse.

Begin with static tiered routing on the obvious wins. Find the high-volume, low-difficulty step types, classification, routing, extraction, formatting, and route them down. These are safe, deliberate, and capture most of the margin. Don't get clever yet.

Add a deterministic verifier wherever you can. The cheapest, most reliable escalation signal isn't model confidence, it's a hard check. Did the JSON validate? Is the extracted value in the allowed set? Does the tool call have required fields? Deterministic verifiers let you route aggressively and catch failures without paying for a second model to judge the first.

Hold the quality floor as a hard constraint, not a soft goal. Define the minimum acceptable quality per task type and treat any routing change that drops below it as a failed change, full stop. Margin is the optimization target; quality is the constraint you optimize within. Teams that invert this, chasing cost with quality as a nice-to-have, are the ones that churn customers.

Re-tune on a schedule, because the ground keeps moving. New model versions ship constantly, and the cheap models are improving faster than the expensive ones. A step that needed the flagship six months ago might run fine on a mid-tier model today, which is pure margin you'll miss if your routing is frozen. This connects directly to the broader pricing-as-model-costs-drop problem the whole category is wrestling with, falling inference costs are a margin gift, but only if your architecture is positioned to capture them.

Routing and Your Pricing Model Are Coupled

You can't think about routing in isolation from how you charge, because the two together determine whether margin gains actually reach your bottom line.

Under flat or subscription pricing, routing savings flow straight to margin, every cheap token is money you keep. Under pure outcome-based or per-resolution pricing, routing is what makes the unit economics survivable in the first place; the price is fixed by the value delivered, so your only path to margin is driving cost down underneath it. This is precisely why aggressive routing and outcome pricing tend to show up together. Under metered, pass-through token pricing, where you bill the customer for tokens consumed, routing is more complicated, because savings you generate flow partly to the customer, not you, unless you've structured a markup that lets you keep some of the delta.

The strategic point is that routing is not just a cost optimization sitting in your infrastructure. It's a pricing weapon. A vendor with a disciplined router can profitably offer outcome-based pricing that a competitor running the flagship on every step simply cannot match without bleeding. Margin earned through routing can be partially handed back as lower, more competitive prices, turning an infrastructure advantage into a go-to-market one. The vendors that win the next two years of GaaS won't be the ones with the best model. They'll be the ones who deploy the right model for each step and keep the savings.

Insights Most People Overlook

The cheap model improving is a bigger deal than the expensive model improving. Industry attention fixates on frontier-model releases, but for a GaaS operator's margin, the steady improvement of small and mid-tier models matters more. Every time the cheap tier gets good enough to absorb a step that previously required the flagship, your blended cost drops with zero price change. The competitive edge in GaaS is increasingly about how quickly you re-route work downward as the budget tier improves, a muscle most teams don't have.

Routing creates a hidden quality-attribution problem in your support data. When something goes wrong, you often can't tell whether the agent failed because the task was genuinely hard or because your router sent it to a model that wasn't up to it. Without logging which model handled which step, every quality incident is unattributable, and you'll waste cycles "fixing" the agent when the real bug was a routing decision. Log the route on every step or you're debugging blind.

The escalation threshold is a business decision masquerading as an engineering parameter. Where you set the cascade's confidence cutoff directly trades margin against quality, and that tradeoff should be owned by whoever owns the P&L, not buried in a config file an engineer tuned once. A threshold that's "too conservative" is leaving margin on the table; "too aggressive" is silently shipping worse outcomes. This is a pricing-and-product decision, and it deserves to be reviewed like one.

Routing can quietly violate your own pricing promises. If you've marketed "powered by [flagship model]" and your router sends 70% of work to a cheaper model, you have a positioning problem the moment a sophisticated buyer audits it. The cost savings are real, but so is the gap between what you sold and what you ship. The vendors handling this well stop selling the model name and sell the outcome, which is where the category is heading anyway.

The biggest margin leak isn't the model tier, it's redundant calls. Teams obsess over cheap-vs-expensive while their agent re-reads the same context five times, re-runs steps that could be cached, and retries needlessly. Deduplication, prompt caching, and cutting unnecessary steps often deliver more margin than model routing, and they carry zero quality risk. Route, yes, but kill the redundant calls first.

References

• Anthropic, Model Pricing
• OpenAI, API Pricing
• Chen, Zaharia & Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (arXiv)
• McKinsey, The Economic Potential of Generative AI: The Next Productivity Frontier