Evaluating agent tool calls: outcomes, paths and consistency

Most teams test their first agent the same way. Someone types a handful of requests into a chat window, watches the tool calls scroll by, and decides it "seems fine." That works right up until a prompt tweak or a model upgrade quietly breaks something nobody re-typed. Anthropic's engineering team describes the breaking point well in Demystifying evals for AI agents (January 9, 2026): users report that the agent "feels worse after changes," and the team is "flying blind" with no way to check except guess and check. This article is about getting out of that loop for tool-using agents specifically. It covers what to grade, when the sequence of tool calls matters and when it does not, how to handle the fact that the same agent passes and fails the same task on different runs, and the grading mistakes that make a good agent look broken.

If you have read our posts on writing tool descriptions and schemas and handling tool errors in agent loops, think of this as the piece that tells you whether those changes actually helped.

Get the vocabulary straight first

Agent evals get confusing fast because "the test passed" can mean several different things. The Anthropic post offers a set of definitions worth adopting as-is. A task is a single test with defined inputs and success criteria. Each attempt at a task is a trial, and because model outputs vary between runs, you run several. A grader is the logic that scores some aspect of a trial, and one task can have several graders. The transcript, which other tools call a trace or trajectory, is the complete record of a trial: outputs, tool calls, reasoning and intermediate results. The outcome is the final state of the environment when the trial ends.

The difference between the last two is the most important idea in the whole field. Anthropic's example is a flight-booking agent that ends its transcript with "Your flight has been booked." The outcome is whether a reservation actually exists in the environment's SQL database. An agent can say the right thing and do the wrong thing, and an eval that only reads the final message will never notice. OpenAI's trace grading guide makes the complementary point from the other side: grading the trace, "the end-to-end log of decisions, tool calls, and reasoning steps," tells you why an agent succeeded or failed rather than only whether it did.

Grade the outcome before you grade the path

The most rigorous academic version of outcome grading is τ-bench, a benchmark from 2024 in which a language model simulates a customer while the agent under test works through domain tools and a written policy, in settings like retail and airline support. Its evaluation "compares the database state at the end of a conversation with the annotated goal state." It does not care what the agent said along the way. Did the order get exchanged, with the right item, and nothing else changed? That is the question.

You can borrow this design directly even for a small agent. Suppose you run a support agent with three tools: lookup_order, issue_refund and send_email. Instead of asserting on the agent's reply, run each task against a test database seeded with known orders, then check rows afterward. A refund task passes when the refunds table has exactly one new row for the right order and amount. A task where the customer asks for a refund outside the policy window passes when the refunds table is unchanged. The second kind of task matters as much as the first, and we will come back to it.

Anthropic's post goes further and recommends a specific order of preference for graders: "deterministic graders where possible, LLM graders where necessary or for additional flexibility," and human graders "judiciously for additional validation." Code-based checks are fast, cheap, objective and reproducible, in the post's words, while their weakness is being brittle to valid variations. Model-based graders handle open-ended output but are non-deterministic and "require calibration with human graders for accuracy." For tool-using agents, much of what you care about is checkable in code: rows in a table, files on disk, a status field on a ticket. Use an LLM judge for the parts that genuinely need judgment, such as whether the email the agent sent was clear and polite.

When the sequence of tool calls does matter

There is a natural instinct to write tests like "the agent must call lookup_order, then issue_refund, then send_email." Anthropic warns against making that the main check. It found that asserting a fixed sequence of tool calls is "too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate," and its advice is that "it's often better to grade what the agent produced, not the path it took."

That does not mean the path never matters. Some tool calls are requirements in their own right. A support agent should verify identity before touching an account; a coding agent should run the tests before declaring victory. Anthropic's own illustrative refund task includes a tool-call grader that requires verify_identity, a process_refund call with an amount of at most 100, and send_confirmation, alongside a state check and a rubric. The trick is to check only the calls that are genuinely required and to leave the rest flexible.

LangChain's open-source agentevals package is a good way to see what "flexible" can mean in code. Its trajectory match evaluator has four modes. "Strict" requires the same messages in the same order with the same tool calls. "Unordered" requires the same tool calls in any order, which the README describes as useful when you want to allow flexibility in how an agent gathers information "but still do care that all information was retrieved." "Subset" and "superset" match partial trajectories against a reference. Separately, a tool-argument match mode controls how arguments are compared: by default only calls to the same tool with the same arguments count as equal, but you can ignore arguments, accept a subset or superset of them, or pass per-tool overrides, including a custom comparator function. The README's own example makes a get_weather call match regardless of the city name's capitalization.

In practice, superset mode with a short reference list is often the right default for safety-relevant calls. It reads as "whatever else you did, you must have done these," which is exactly the shape of a rule like "always check identity before a refund."

Run every task more than once

Here is the part that surprises people coming from ordinary unit tests. The same agent, on the same task, with the same prompt, will sometimes pass and sometimes fail. Anthropic suggests two numbers to describe this. pass@k is the probability of at least one success in k trials, which rises as k grows. pass^k is the probability that all k trials succeed, which falls as k grows. The post's arithmetic example: an agent with a 75% per-trial success rate has about a 42% chance of passing three trials in a row, since 0.75 cubed is roughly 0.42. At k equals 1 the two numbers are identical, and by k equals 10 they "tell opposite stories."

τ-bench introduced pass^k precisely because single-run scores hide this. Its abstract reports that even state-of-the-art function-calling agents at the time, such as GPT-4o, succeeded on fewer than half of its tasks, and were "quite inconsistent," with pass^8 below 25% in the retail domain. For an agent that touches money or customer records, consistency is the property you actually ship. A user does not experience your average; they experience the one run where the agent refunded the wrong order.

Computing both is a few lines once you store per-task results. If a task passed in c out of n trials, a simple estimate of its per-trial success rate is c / n, and you can report the mean across tasks for pass@1 alongside the share of tasks that passed every trial. The second number is the one to watch when you change a prompt or swap models, because a change can leave the average flat while making the agent less reliable.

Build the set from real failures, and test both directions

The most common reason teams never write evals is the belief that they need hundreds of tasks. Anthropic's advice is blunt: "20-50 simple tasks drawn from real failures is a great start." Early in a project, each change has a large, visible effect, so a small set is enough to detect it. Start with the checks you already do by hand before each release, then mine your bug tracker and support queue. Every reported failure is a free test case with a known correct answer.

Two rules keep that set honest. First, every task should be unambiguous. The post's test is whether two domain experts would independently reach the same pass or fail verdict, and it recommends writing a reference solution that passes all graders, which proves the task is solvable and that the graders are wired correctly. Second, the set should be balanced. Anthropic describes building web search evals for Claude.ai in both directions: queries where the model should search, like the weather, and queries it should answer from knowledge, like "who founded Apple?" Testing only one direction, the post says, produces one-sided optimization, such as an agent that searches for almost everything. For a tool-using agent this means every "should call issue_refund" task needs a sibling where calling it would be wrong.

The approval gates we covered yesterday are a good source of these negative cases. Any call you would gate in production deserves at least one eval task where the correct behavior is not to make the call at all.

Isolate trials, and suspect the grader first

Agent evals have a failure mode ordinary tests rarely do: the environment leaks. Anthropic recommends starting each trial from a clean environment, because leftover files, cached data or exhausted resources cause correlated failures that look like agent problems. Shared state can also inflate scores. In one internal case, the post notes, Claude gained an unfair advantage on some tasks by examining the git history left by previous trials. If your agent can read anything a previous run wrote, reset it.

Then there are grader bugs, which are more common than anyone expects. The post's most striking example is CORE-Bench, where Opus 4.5 first scored 42%. An Anthropic researcher found rigid grading that rejected "96.12" when the expected answer was "96.124991…", along with ambiguous task specs and tasks that could not be reproduced exactly. After the fixes, and with a less constrained scaffold, the score jumped to 95%. The general rule from the same post: with frontier models, a 0% pass rate across many trials is "most often a signal of a broken task, not an incapable agent."

For LLM judges, two practical tips from the post are worth copying. Give the judge a way out, such as an instruction to return "Unknown" when it lacks information, so it does not guess. And grade each rubric dimension with a separate, isolated judge rather than asking one prompt to score everything at once.

Split capability evals from regression evals

Finally, keep two suites with different goals. Anthropic distinguishes capability evals, which ask what the agent can do and should start with a low pass rate, from regression evals, which ask whether it still handles everything it used to and should sit near 100%. As capability tasks become reliably solved, they "graduate" into the regression suite. Run the regression suite on every prompt or model change; treat any drop as a bug.

None of this replaces reading transcripts. The Anthropic post's own summary ends with "Read the transcripts!", and its reasoning holds for any agent: when a task fails, only the transcript tells you whether the agent made a genuine mistake or your grader rejected a valid solution. Pick a weekly slot, read a sample of failed and passed runs, and fix the eval whenever a failure looks unfair. An eval suite you trust is what lets you try a cheaper model from our model comparison on a Tuesday and know by Wednesday whether it is safe to switch.

Anthropic's short list of twenty to fifty tasks is the right place to start. For a deeper systems framing of evaluation, datasets and production monitoring around LLM applications, AI Engineering by Chip Huyen is a solid next read after you have a trusted suite running.

If you run models on your own machine, we keep a short list of local LLM gear — GPUs, Macs, RAM and storage — with plain-text Amazon links.