Batch APIs for agent workloads: when half price is worth the wait
OpenAI, Anthropic and Google all sell batch inference at 50% off. Here is which agent jobs fit a 24-hour window, which do not, and the details that bite.
Most of the cost advice for agent builders is about squeezing tokens: shorter prompts, smarter caching, tighter tool results. Those are worth doing, and we have covered them in our guides to tool-calling costs and context compaction. But there is a simpler lever that a lot of teams leave untouched, because it looks like it belongs to data engineers rather than agent builders. All three big model providers will run your requests at half price if you are willing to wait for the answers. This post looks at what those batch APIs actually promise, which parts of an agent system can live with the wait, and the small details in each provider's documentation that tend to cause trouble the first time you use them.
The offer, provider by provider
The headline number is the same everywhere. OpenAI's Batch API guide describes a "50% cost discount compared to synchronous APIs," a separate pool of higher rate limits, and a completion window of 24 hours, "and often more quickly." Anthropic's Message Batches documentation says that "all usage is charged at 50 percent of the standard API prices," with most batches finishing in less than an hour. Google's Gemini Batch API page prices batch usage at 50% of the standard interactive cost, with a target turnaround of 24 hours that it says is usually much quicker.
To make that concrete, Anthropic's batch pricing table lists Claude Haiku 5.5 at $0.05 per million input tokens and $0.25 per million output tokens for prompts up to 100,000 tokens. Our model data has the standard price at $0.10 and $0.50, so the table really is half. Suppose you run a nightly evaluation suite of 2,000 test cases, each with about 6,000 input tokens and 800 output tokens. That is 12 million input tokens and 1.6 million output tokens. At standard Haiku 5.5 prices the run costs about $2.00; as a batch it costs about $1.00. That arithmetic is ours and the numbers are illustrative, but the shape holds for any model: if a job is big and nobody is waiting on it, you are paying double for speed you do not use.
The limits differ more than the prices. Anthropic caps a single batch at 100,000 requests or 256 MB, whichever comes first, and keeps results available for 29 days. OpenAI caps a batch at 50,000 requests and a 200 MB input file, and limits you to 2,000 batch creations per hour. Google accepts inline requests when the whole batch stays under 20 MB, and a JSON Lines input file of up to 2 GB for anything bigger, with results kept for six weeks. On OpenAI the batch pool is separate from your normal per-model limits, so the guide notes that batch work will not consume tokens from your standard rate limits. That matters as much as the discount for teams whose daytime agent traffic already runs close to its ceiling.
Why an agent loop and a batch do not mix directly
Here is the catch for agent builders. A batch request is one model call. An agent is a loop of model calls, where each turn depends on the result of the tools the previous turn asked for. If you put an agent's first turn in a batch, you might get back a response asking to call a tool, and the next turn cannot start until you run that tool and submit another request. With a window of up to 24 hours per round trip, a ten-step task could in the worst case take days. Batch APIs are built for breadth, many independent calls at once, not for depth.
That does not make tool use unsupported. Anthropic's list of what can be batched explicitly includes tool use, multi-turn conversations, extended thinking and its server tools such as web search, web fetch and code execution. What it means is that the unit you put in a batch should be a self-contained step. A request with the full conversation so far is fine, and so is a request that uses a server-side tool. A design where your own code has to answer a tool call before the model can continue is a poor fit unless you can afford to advance thousands of conversations together, one turn per batch, which is a pattern some offline simulation and data-generation jobs use deliberately.
The agent work that does fit
Once you think of a batch as a pile of independent steps, a surprising amount of an agent system qualifies. The clearest case is evaluation. Our post on evaluating agent tool calls argues for running the same test cases many times to measure consistency. Those runs are exactly the kind of large, parallel, nobody-is-watching work that batch pricing rewards, and an LLM-as-judge pass over the transcripts afterward is an even better fit, because every grading call is a single request.
The second case is enrichment and preprocessing. Agents are often only as good as the material they retrieve, and preparing that material is usually a one-shot model call per item: summarizing documents for a search index, extracting fields from support tickets, tagging a product catalog, or writing the short descriptions a retrieval tool will return. OpenAI's batch endpoints include /v1/embeddings alongside /v1/responses and /v1/chat/completions, so a nightly reindex can run both the summaries and the embeddings at the discounted rate.
The third case is anything your users already expect to be slow. A weekly report an agent writes for each customer, a backlog of documents to classify, or a research digest that lands in an inbox the next morning do not need a synchronous call. If the product promise is "by tomorrow," the cost structure can be too. The test is simple: if a person or another process is blocked until the answer arrives, keep it synchronous; if not, it is a batch candidate.
Batches and caching stack, with caveats
Prompt caching and batch discounts are not either-or. Anthropic's documentation says the two discounts "can stack," but warns that because batch requests run asynchronously and concurrently, cache hits are best effort, with customers typically seeing hit rates between 30 and 98 percent depending on traffic patterns. Its advice is to put identical cache_control blocks in every request, to share as much cached content as possible, and, because batches can take longer than five minutes, to consider the one-hour cache duration. Google's page says context caching works for batch requests too, when each request names the cached_content resource, and that a cache hit is billed at the normal context caching rates.
For an evaluation suite this is a natural match. Every test case shares the same long system prompt and tool definitions, and only the user turn differs. Put the shared part first, mark it for caching, and keep the variable part at the end.
The details that bite
Order is the first trap. OpenAI states plainly that "the output line order may not match the input line order," and Anthropic's best practices say the same, so every request needs a meaningful custom_id. Anthropic restricts that ID to 1 to 64 letters, digits, hyphens and underscores. Make it something you can join back to your own records, such as a test case ID plus a run number, rather than a counter that only makes sense if the order holds.
Partial results are the second. A batch is not all-or-nothing. Anthropic reports each request as succeeded, errored, canceled or expired, and says you are not billed for the last three. OpenAI marks a batch expired if it could not finish within the 24-hour window, and Google moves a job to JOB_STATE_EXPIRED if it has been running or pending for more than 48 hours, in which case there are no results to retrieve. Your collector should read results per request, retry what failed in a new batch, and never assume that a finished batch means every line has an answer. The same retry discipline we described in handling tool errors applies here, just on a slower clock.
Duplicates are the third. Google's best practices warn that creating a batch job is not idempotent: send the same creation request twice and you get two jobs and two bills. Record the job name the moment you get it, and have your submit step check for an existing job before creating another one.
Validation is the last. Each provider rejects some parameters inside a batch. OpenAI's batch worker rejects requests that set stream=true, and Anthropic returns validation errors for streaming, its fast mode speed setting, server-side fallbacks and max_tokens: 0. Anthropic also notes that busy batches can go slightly over a workspace spend limit. The cheapest protection is the one its docs suggest: dry run a single request shape against the normal synchronous API first, then submit the batch.
A practical way to start
You do not need to redesign anything to try this. Pick one job in your system that already runs on a schedule, most often the eval suite or a nightly enrichment step. Change it to write its requests to a JSON Lines file with stable IDs, submit the file, poll on a slow timer, and join the results back by ID. Run it next to the synchronous version for a few nights and compare the outputs and the bill. If the answers match and nobody noticed the delay, you have cut that line item in half with a change that lives entirely in the plumbing, and you can look for the next scheduled job to move.