Running an agent on a local model: the tool-calling settings that quietly break it
Ollama, vLLM and llama.cpp all speak OpenAI-style tool calling, but each has a default that can wreck an agent loop, from a 4k context window to unconstrained arguments. Here is what to change before you blame the model.
Some links are affiliate links; we may earn a commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases. Disclosure
Pointing an agent at a local open-weight model looks like a one-line change. Ollama, vLLM and llama.cpp's llama-server all expose an OpenAI-compatible chat endpoint, all accept a tools array, and all hand back tool_calls in the shape your loop already parses. So you swap the base URL, run your agent, and watch it forget its instructions by turn three, call a tool with half its arguments, or answer in prose when it should have called anything at all.
Most of the time the model is not the problem. Each of these servers ships with defaults that are sensible for a chat window and wrong for an agent, and the documentation says so if you know where to look. This article goes through the settings that matter on each server, quoting the docs directly, so you can rule out configuration before you start comparing models.
The context window you get is not the one the model supports
The first trap is the most common one, and on Ollama it is documented in plain sight. The Ollama context length page says the server picks a default based on your GPU memory: a 4k context below 24 GiB of VRAM, 32k between 24 and 48 GiB, and 256k at 48 GiB or more. The same page adds that "tasks which require large context like web search, agents, and coding tools should be set to at least 64000 tokens."
Think about what 4,000 tokens means for an agent. A system prompt plus a handful of tool definitions with real descriptions and JSON schemas can easily take a couple of thousand tokens before the user has said anything. Add one tool result, such as a page of search results or a file listing, and the window is full. From there something has to give, and if what drops out of view is the instructions or tool definitions, the model behaves like one that was never told about its tools.
The fix is one environment variable. The docs show OLLAMA_CONTEXT_LENGTH=64000 ollama serve, or you can use the slider in the Ollama app's settings. Then confirm it took: ollama ps prints a CONTEXT column next to each loaded model, along with a PROCESSOR column that shows how much of the model is on the GPU. The docs recommend avoiding offloading the model to CPU, and warn that a larger context needs more memory, so raising the window on a small card can push part of the model off the GPU and slow every turn. If your agent was fine at 4k in a quick test and falls apart on real tasks, check this before anything else.
The lesson carries over to any local server: find out the context the server actually allocated, not the number on the model card, and size it for your longest realistic run.
A tool call that parses is not the same as a good one
Some hosted APIs offer a strict mode that holds arguments to your schema. Local servers can do something similar, but it is not always on, and the guarantee is narrower than people assume.
The vLLM tool calling documentation is unusually candid about this. To let the model decide on its own when to call tools, you start the server with --enable-auto-tool-choice and a --tool-call-parser that matches the model family. In that mode, unless at least one tool sets strict: true or the server raises the floor with --tool-strict-level, "vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema." In other words, the default auto mode is a parser reading the model's output after the fact, not a grammar holding it to your schema.
Forcing a call changes that. With a named function or tool_choice="required", vLLM uses its structured outputs backend to constrain generation, and the docs promise that with required "the model is guaranteed to generate one or more tool calls." Even here they draw a careful line: for named calls, "you are guaranteed a validly-parsable function call - not a high-quality one." The JSON will parse. Whether the city is the right city is still up to the model. The docs also warn that the first named call on a server can take several seconds or more while the grammar compiles before it is cached, which is worth knowing before you mistake a cold start for a hung agent.
Strictness in vLLM also works per tool. The docs explain that the argument schema of a tool is pinned only when that tool sets strict: true, or when the server runs with --tool-strict-level parameter, and that a tool that omits strict "keeps its arguments unconstrained, even when another tool in the same request is strict." If you add strict: true to one tool to test it, the rest of your tools are still running on raw text extraction. For an agent, setting the server-wide level is usually the safer default, and you can still let the model answer in plain text, since the docs note that with tool_choice="auto" the grammar does not force a tool call.
Ollama: what the OpenAI-compatible endpoint leaves out
Ollama's native tool calling guide covers single calls, parallel calls and streaming. The streaming section has a detail that trips up hand-rolled loops: you need to "gather every chunk of thinking, content, and tool_calls, then return those fields together with any tool results in the follow-up request." If your loop only keeps the text content from a streamed turn, the next request is missing the very tool call the result is supposed to answer.
If you reach Ollama through its OpenAI-compatible endpoint instead, read the field list on the OpenAI compatibility page. Under /v1/chat/completions, tools is supported but tool_choice is unchecked. Any part of your agent that relies on forcing a call, such as making the first step always a search or ending with a submit_answer tool, will not get that behavior there. The request still succeeds, and the model is simply free to ignore the instruction, which makes the bug easy to miss. Either move that step's control into your own code, by validating the reply and retrying when the expected call is missing, or use a server that honors tool_choice.
llama.cpp: templates, parallel calls and the KV cache
The llama.cpp function calling doc starts with a requirement that is easy to skip: tool calling in llama-server works when the server is started with the --jinja flag. It then explains that the server has native handlers for a list of model families' tool-call formats and falls back to a generic handler when it does not recognize the chat template, which you can spot as Chat format: Generic in the logs. The doc warns that generic support "may consume more tokens and be less efficient than a model's native format," and suggests overriding the template with --chat-template-file when appropriate. If your local model's tool calls look clumsy, check the log line first, because a missing or unrecognized template means the model is being prompted in a format it was not trained on.
Two more lines in that doc matter for agents. Parallel tool calling "is supported on some models but disabled by default," and you turn it on per request with "parallel_tool_calls": true. If you built your loop against a hosted API that batches independent calls, expect more turns locally until you set that. And there is a caution box that is worth repeating word for word: "Beware of extreme KV quantizations (e.g. -ctk q4_0), they can substantially degrade the model's tool calling performance." Quantizing the key-value cache is a popular way to fit a longer context into limited memory, which is exactly what the Ollama section above pushes you toward. The two settings pull against each other, and tool calling is what suffers if you go too far.
Judging the model after the settings are right
Once the context is sized, the template is recognized and arguments are constrained, you can finally compare models fairly. The Berkeley Function Calling Leaderboard is the usual public reference. Its fourth version describes itself as moving "from tool use to agentic evaluation," covers multi-turn interactions, and separately labels models that use native function calling ("FC") and ones driven through a prompt-based workaround ("Prompt"). That split mirrors the native and generic handlers in llama.cpp, so check which mode a score came from before you assume it matches your setup. The page also states when it was last updated and which evaluation commit it used, which helps when you are reading it months later.
A leaderboard still won't tell you how a model handles your tools. The most useful test is your own: take twenty or so real tasks from your agent's logs, run them through the local setup with the settings above, and count how often the tool calls are correct, not just parseable. We covered how to build that kind of check in evaluating agent tool calls, and the same suite works for comparing a local model against a hosted one.
Hardware is part of the configuration
Everything above comes back to memory. Ollama's own default thresholds sit at 24 and 48 GiB of VRAM, and the context you can afford, the cache quantization you can avoid and whether the whole model stays on the GPU all depend on how much you have. If you are deciding what to run local agents on, a card in the 24GB VRAM GPU range is the point where Ollama stops defaulting you to a 4k window, though you should size it against the models you actually plan to run.
A short checklist before you blame the model
Before you write a local model off as bad at tools, set the context to at least the 64,000 tokens Ollama recommends for agents and confirm it with ollama ps. On vLLM, use a parser that matches the model and turn on strict schemas server-wide rather than relying on raw text extraction. On llama.cpp, start with --jinja, check that the log does not say Chat format: Generic for a model that has a native format, enable parallel calls if your loop expects them, and go easy on KV cache quantization. On Ollama's OpenAI-compatible endpoint, do not count on tool_choice. Then run your own tasks and count correct calls. If the model still struggles after all that, at least you will know it is the model.