Model routing for agents: when sending easy work to a cheap model actually cuts the bill

The price gap between models is now wide enough that model choice is the biggest cost lever most agent builders have. On Anthropic's pricing page, Claude Sonnet 5.5 lists at $2 per million input tokens and $10 per million output tokens, while Claude Haiku 5.5 lists at $0.10 and $0.50 for prompts up to 100,000 tokens. That is a twenty-fold difference on both sides. OpenAI's pricing page shows the same shape: GPT-6.1 Sol at $2 and $10, GPT-6 Luna at $0.10 and $0.50. If even half of your agent's work could run on the small model without hurting results, the savings would dwarf anything you get from trimming prompts.

The idea of sending each request to the cheapest model that can handle it is called routing, and it is not new. What is new is how it interacts with the way agents actually spend money: long multi-turn loops, growing histories, and prompt caches that only pay off if the model stays the same. This article walks through three routing patterns, prices each one on a concrete agent run, and explains why the most intuitive version, switching models step by step inside a loop, is often the worst.

What the research says routing can save

Two papers set the baseline. FrugalGPT, from Stanford in 2023, surveyed API prices that "can differ by two orders of magnitude" and proposed an LLM cascade: ask a cheap model first, score the answer, and only escalate to an expensive model when the score is too low. On its benchmarks the authors report matching the best single model "with up to 98% cost reduction." RouteLLM, from the LMSYS team in 2024, took the other approach: train a small router that looks at the query up front and picks a strong or weak model before anything runs. The RouteLLM write-up reports cost reductions "of over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K" compared with using only GPT-4, while keeping 95 percent of GPT-4's quality, and the code is open source in the RouteLLM repository.

Those numbers are real, but read them carefully. Both papers measured single-turn questions with a clear right answer, routed between models from 2023 and 2024. The savings swing from 35 to 85 percent depending on the benchmark, which tells you the result depends heavily on how much of your traffic is genuinely easy. Neither paper priced a forty-turn agent with tool calls and a warm prompt cache. For that, you need to do the math on your own loop.

Anthropic's own guide to building effective agents lists routing as a core workflow pattern and gives exactly this example: sending "easy/common questions to smaller, cost-efficient models" and "hard/unusual questions to more capable models." It adds the condition that matters most, that routing works "where classification can be handled accurately." A router that guesses wrong sends hard work to a model that fails at it, and a failed agent run is the most expensive kind.

A run to price

Take a support agent that resolves one ticket per run. Its fixed prefix, meaning the system prompt plus tool definitions, is 8,000 tokens. Each turn adds about 2,000 tokens of history (the model's tool call, the tool result, and a little reasoning), and the model writes about 300 output tokens per turn. A typical run takes 20 turns, so the last request carries 46,000 tokens, and the run sends 540,000 input tokens in total across all its requests.

With prompt caching on, almost all of that input is a cache read. Each token is written to the cache once, which comes to 46,000 written tokens, and the remaining 494,000 are read back on later turns. On Sonnet 5.5, Anthropic's prompt caching documentation lists five-minute cache writes at $2.50 per million and cache hits at $0.10 per million. That makes the run cost about $0.115 in cache writes, $0.049 in cache reads and $0.06 in output, roughly $0.22 in all.

The same run on Haiku 5.5, which the pricing page describes as built "for high-volume, latency-sensitive tasks such as classification, extraction, and routing," costs $0.125 per million for cache writes, $0.01 for cache hits and $0.50 for output. That works out to about $0.014 per run, roughly sixteen times cheaper. The question is how much of your traffic can take that discount without failing.

Pattern one: route the whole run up front

The simplest pattern, and usually the best one for agents, is to classify the task before the loop starts and run the entire loop on one model. A small model reads the ticket and answers one question, such as "is this a routine password reset or order lookup, or something unusual?" With a 500-token instruction, a 1,500-token ticket and a ten-token answer, that classification costs about $0.0002 on Haiku 5.5, which is noise.

Suppose 70 percent of tickets are routine. The blended cost is 0.7 times $0.014 plus 0.3 times $0.22 plus the classifier, about $0.077 per ticket. Against $0.22 for an all-Sonnet agent, that is about 65 percent cheaper, or roughly $770 instead of $2,240 per 10,000 tickets. The model never changes inside a run, so the prompt cache works exactly as it would for a single-model agent.

The catch is the 30 percent. If the classifier marks a hard ticket as routine, the Haiku run may loop, call the wrong tools or give a confident wrong answer. You need a way to notice that and recover, which leads to the second pattern.

Pattern two: cascade, then escalate

A cascade skips the up-front decision. Every task starts on the cheap model, and a check at the end decides whether the result is good enough. If it fails, the task is rerun on the strong model. That check can be a programmatic test (did the refund total match the order, did the code compile), a schema check on the final answer, or a judge model, and our guide to evaluating agent tool calls covers how to build one you can trust.

Priced on our run, a cascade where 30 percent of tasks fail the check costs $0.014 for every Haiku attempt plus 0.3 times $0.22 for the reruns, about $0.081 per task. That is slightly more than the up-front router, because the failed tasks pay for both runs, but it needs no classifier and it catches the misroutes a classifier would miss.

The real limit on cascades in agents is side effects. FrugalGPT could throw away a bad answer for free because a bad answer was just text. A bad agent run may have already sent an email, issued a refund or written to a database. Before you rerun, you need the run to have been read-only, or to have used idempotent tool calls and approval gates on anything irreversible. Our posts on handling tool errors and approval gates for tool calls cover both. A practical compromise is to cascade only the planning or research phase, where tools only read, and commit to one model before the first write.

Pattern three: switching models mid-run, and why it costs more than it looks

The tempting version of routing is per step: let the cheap model handle the routine lookups and hand the tricky step, such as a refund decision, to the strong model, then hand back. The per-token prices make it look like a clear win. The prompt cache says otherwise.

A cache entry belongs to the model that wrote it. OpenAI's prompt caching guide lists model first among the request settings that affect whether a later request can reuse a cached prefix, noting that "a different model can use different weights and caching behavior." In practice, treat every model switch as a full cache miss on the new model.

Price it on our run. Suppose the agent runs on Haiku 5.5 for ten turns and then switches to Sonnet 5.5 for the hard part. Turn 11 carries 28,000 tokens of prompt, and Sonnet has never seen any of it, so all 28,000 are written to its cache at $2.50 per million, about $0.07. Had Sonnet been running from the start, it would have read those same tokens from cache for under three-tenths of a cent. One switch costs about a third of what the entire all-Sonnet run costs, before Sonnet has done any work. Switch back to Haiku and you may pay again if its cache has expired, and an agent that bounces between models every few turns can end up costing more than one that never left the strong model.

There are two ways to keep the benefit. The first is to switch at most once, in one direction, at a natural phase boundary, and to do it early, while the history is still short. The second, usually better, is not to hand over the history at all. Spin up the strong model as a subagent with a short briefing: the ticket, the facts the cheap model gathered, and the decision it must make. A 6,000-token briefing costs about $0.015 to write on Sonnet 5.5, a fifth of the switch cost, and the strong model reasons over a clean context instead of twenty turns of tool noise. Our post on agent memory design covers what belongs in a briefing like that.

Two price traps to check before you route

Small models are not always flat-priced. Anthropic's pricing page says Haiku 5.5 "is priced by prompt length": prompts over 100,000 tokens pay $0.50 input and $2.50 output, five times the short-prompt rate, and the threshold counts cache reads and writes too. A long-running Haiku agent that drifts past 100,000 tokens loses much of its advantage, which is one more reason to route whole runs and keep them short, or to pair the cheap model with context compaction.

Cache minimums also differ by model. The Anthropic caching docs list a 512-token minimum cacheable prompt for the current Claude 5.x models, including Haiku 5.5, while OpenAI's guide puts the minimum at 1,024 tokens for GPT-5.6 and later. A router that sends very short prompts to a small model may get no caching at all, which is fine for a one-shot classifier and wasteful for anything that loops.

How to start

Measure before you route. Log every run with its final outcome, its token usage and its cache fields. Anthropic returns cache_creation_input_tokens and cache_read_input_tokens in the response usage, and OpenAI reports cached_tokens under usage.input_tokens_details. Then take a few hundred real tasks, run them on both a small and a large model, and see where the small one fails. If its failures cluster in a recognizable kind of task, a whole-run router is your best bet. If they are scattered and hard to predict, a cascade with a good end-of-run check is safer. If they cluster in one step, hand that step to a strong subagent with a short briefing rather than switching the main loop.

Whatever you choose, re-run the comparison when prices change, because the right split is a function of the price gap. Our model comparison table tracks current list prices, and the token cost estimator is a quick way to sanity-check the per-run math above before you commit to it. For a longer treatment of model selection, evaluation and cost trade-offs in production LLM systems, AI Engineering by Chip Huyen is a solid companion.

Sources

FrugalGPT (arXiv 2305.05176) and RouteLLM (arXiv 2406.18665) papers; the LMSYS RouteLLM blog post and repository; Anthropic's "Building effective agents," pricing page and prompt caching documentation; OpenAI's pricing page and prompt caching guide. All linked above and checked November 1, 2026. Prices match our model table as verified October 29, 2026; all cost figures are our own arithmetic on those list prices.