Parallel tool calls: how OpenAI, Anthropic and Gemini batch them, and how to run the batch safely

Ask an agent "what's the weather in San Francisco and New York, and what time is it in each?" and a well-behaved model will not make four round trips. It will return four tool calls in a single response and wait for all four answers. That behavior, usually called parallel tool calling, is one of the cheapest latency wins in agent design, because every round trip you skip saves a full model turn of waiting and a full re-read of the conversation. It is also a frequent source of quiet bugs, because the model only proposes the batch. Your harness decides how to run it, how to report failures, and how to format the answers, and getting any of those wrong can make the model stop batching or, worse, make it act on results that do not belong to the call it made.

This post compares how OpenAI, Anthropic and Google document the feature today, then covers the execution and history rules that keep it working. Everything below comes from the providers' own documentation and the Python standard library reference, linked inline.

What "parallel" actually means here

The word is a little misleading. None of these APIs runs your functions. The model returns a list of calls in one turn, and that is all the parallelism the provider gives you. Anthropic's parallel tool use page says this plainly: the API "doesn't prescribe an execution order," and you can run the calls concurrently with Promise.all or asyncio.gather, sequentially in the order they appear, or in any mix that suits your tools. Its guidance is that independent, read-only operations are usually safe to run in parallel for lower latency, while tools with side effects, shared state or ordering requirements might be better run one after another.

OpenAI's function calling guide puts the burden on your code in a different way. It says model responses "can include zero, one, or multiple calls," and that "it is best practice to assume there are several." Each call arrives as a function_call item in the response's output array with its own call_id, a name and JSON-encoded arguments, and you answer each one with a function_call_output item that carries the same call_id.

Google's Gemini function calling documentation describes two related patterns and keeps them apart. Parallel function calling is calling multiple functions in a single turn "when they are independent," and compositional function calling is chaining calls in sequence, such as getting a location first and then the weather for that location. Its example asks a model to "Turn this place into a party!" with three tools for a disco ball, music and lights, and prints each function_call step the model returns. Results go back as function_result items matched by call_id.

The distinction between those two Gemini patterns is the most useful mental model for all three providers. A batch in one turn means the model has decided the calls do not depend on each other's results. If call B needs the output of call A, the model should wait for A and ask for B in the next turn. When you see a batch whose calls clearly do depend on each other, that is a tool-design or prompt problem to fix, not something your executor should try to untangle.

Turning it off, and what each switch really does

All three providers batch by default on current models, and two of them document a switch to stop it.

OpenAI uses a top-level request parameter. Setting parallel_tool_calls to false "ensures exactly zero or one tool is called." The guide adds two caveats worth knowing. On supported models beginning with GPT-5, functions can be called in parallel when built-in tools are also available, but built-in tools themselves "cannot be included in a parallel function-call batch." And if you use a fine-tuned model and it calls multiple functions in one turn, strict mode is disabled for those calls, so the schema guarantee you may be relying on quietly does not apply. The guide also notes that one older snapshot, gpt-4.1-nano-2025-04-14, can sometimes include multiple calls for the same tool when parallel calls are enabled, and recommends disabling the feature for that snapshot.

Anthropic puts the switch somewhere else. Its page says disable_parallel_tool_use: true goes inside the tool_choice object and "is not a top-level request parameter," which is an easy mistake to make if you are porting code from OpenAI. The effect also depends on the tool_choice type. With auto, the default, Claude calls at most one tool per response and can still answer in plain text. With any or tool, it calls exactly one tool. There is a model-specific catch: the same page says Claude Opus 5.5, Claude Sonnet 5.5, Claude Fable 5.1 and Claude Mythos 5.1 do not support the any or tool choice types, so on those models the "exactly one" mode is not available and auto with the flag is the option you have.

Gemini's page documents tool_choice modes in generation_config rather than a dedicated parallel switch. The modes are auto, the default, where the model decides whether to call a function, any, which constrains it to always predict a function call, none, which prohibits calls, and validated, which makes the model ensure schema adherence. The parallel example on that page uses any. We did not find a documented flag on that page for limiting a turn to a single call, so if you need strictly one call at a time on Gemini, the safe approach is to enforce it in your harness by executing only the first call and returning an error result for the rest, which the next section explains how to do cleanly.

Why would you turn batching off at all? The common reason is a tool with side effects where order matters, such as "create the branch" followed by "push the commit," or a payment flow where you never want two charges proposed at once. In those cases a single call per turn gives you a natural checkpoint, which pairs well with the approval gates we covered earlier. For read-heavy agents, such as research, retrieval or code search, leaving it on is almost always the better choice.

Getting the model to batch more

Sometimes the problem is the opposite: the model makes one call per turn when it could have made five. Anthropic's page says Claude 4 and later models make parallel calls by default when a request benefits from multiple tools, and it suggests a system prompt line for pushing harder: "For maximum efficiency, whenever you need to perform multiple independent operations, invoke all relevant tools simultaneously rather than sequentially." It also offers a stronger version that tells the model, for example, to read three files with three parallel calls. The page flags one model in particular: Claude Fable 5.1 "may issue fewer parallel tool calls than earlier models," most noticeably in long agent loops where the next reads are only implied, such as custom coding agents and computer use, while standard function calling is unaffected.

The same page gives a simple way to check whether any of this is working. Across a run, count the responses that contain at least one tool call and the total number of calls in them, then divide. If the average tools per message is not above 1.0, you are not getting parallel calls. That is a cheap metric to log alongside the tool-call evaluations you may already run, and it turns "the agent feels slow" into a number you can watch after a model upgrade.

Running the batch: concurrency without losing errors

Once you have a batch, the obvious code is asyncio.gather over all the calls. The Python asyncio documentation explains why the default form of that is risky inside an agent. With return_exceptions left at False, "the first raised exception is immediately propagated" to the caller, and the other awaitables "won't be cancelled and will continue to run." In an agent, that means one failing lookup can throw you out of the handler while sibling calls keep running, and you lose the results you would need to answer the model.

Two patterns work better. The first is gather with return_exceptions=True, where, in the documentation's words, "exceptions are treated the same as successful results, and aggregated in the result list." You then map each item back to its call and turn exceptions into error results. This fits read-only batches, where one failed search should not stop the others. The second is asyncio.TaskGroup, which the same page describes as giving "stronger safety guarantees than gather": if a task raises, the group cancels the remaining scheduled tasks. That fits batches where a failure means the rest should not happen. Wrap either one in asyncio.timeout() so a hung tool cannot stall the turn forever. The documentation also warns that structured concurrency tools like TaskGroup and timeout() rely on cancellation internally and can misbehave if a coroutine swallows asyncio.CancelledError, which is worth checking in any tool wrapper that catches broad exceptions.

Here is a short sketch of the read-only case, assuming your own run_tool function and a call object with an id, a name and arguments:

async def run_batch(calls, timeout_s=30):
    async with asyncio.timeout(timeout_s):
        outcomes = await asyncio.gather(
            *(run_tool(c.name, c.arguments) for c in calls),
            return_exceptions=True,
        )
    results = []
    for call, outcome in zip(calls, outcomes):
        if isinstance(outcome, Exception):
            results.append({"id": call.id, "is_error": True,
                            "content": f"{call.name} failed: {outcome}"})
        else:
            results.append({"id": call.id, "is_error": False,
                            "content": outcome})
    return results

The key detail is that gather returns results in the same order as the calls you passed, so zipping them back to the calls is safe, and every call gets exactly one result whether it worked or not.

Answering the batch: the history rules that keep batching alive

How you send results back matters as much as how you run the tools. Anthropic is the most prescriptive. Its page says to return one tool_result for each tool_use block, "all together in the next user message," matched by tool_use_id, with every result placed before any text in that message. If you decide not to run a call, for instance because you ran the batch sequentially and an earlier call failed, you should still return a result for it with is_error: true and a short explanation. The troubleshooting section names the most common mistake: sending a separate user message for each tool result. That formatting, the page says, "teaches" Claude to avoid parallel calls, so a harness that splits results can slowly train its own agent to be slower.

The page also calls out a stricter case. For the computer use and browser use tools, when Claude returns several of their actions in one turn, you should run them sequentially in the order they appear and stop at the first failure, returning each tool's defined text for the skipped calls. That is a good default for any tool that changes the state of a screen or a session.

OpenAI and Gemini match results by id rather than by position, through call_id on function_call_output and function_result items respectively. The same discipline applies: one output per call, every id answered, and errors reported as results rather than dropped. A missing answer is the kind of thing that turns into a confusing API error several turns later, and our post on handling tool errors in agent loops covers how to word those error results so the model can recover.

A worked example

Picture a support agent that receives "Has order 4471 shipped, and is the blue jacket in stock in medium?" A good model returns two calls in one turn, get_order_status and check_inventory. Both are reads, so the harness runs them together with gather. The inventory service times out. Rather than failing the turn, the harness returns the order status as a normal result and an error result for inventory that says the service did not respond within 30 seconds. On Anthropic, both results go in one user message ahead of any text. On OpenAI and Gemini, each goes back under its own call_id. The model can then tell the customer the order shipped and offer to check stock again, instead of the whole conversation stalling on one slow dependency.

Now change the request to "cancel order 4471 and refund it." If the model proposes cancel_order and issue_refund in one batch, those are two writes where the second only makes sense if the first succeeded. This is where you either disable parallel calls for that tool set, combine the two into a single tool, as OpenAI's guide suggests for functions that are always called in sequence, or have the harness run the batch sequentially and return an error result for the refund if the cancel fails.

The short version

Parallel tool calling is on by default on current models from all three providers, and it is worth keeping on for read-heavy agents. Know where each switch lives: a top-level parallel_tool_calls on OpenAI, a flag inside tool_choice on Anthropic, and only tool_choice modes on Gemini. Run independent reads concurrently with errors captured, run writes in order, answer every call by its id, and on Anthropic keep all results in one message. If you want to see what a batched turn costs on a given model, our model comparison page lists the official per-token prices we check every day.