Structured outputs for agents: what the JSON schema guarantee actually covers

Every agent eventually has to hand something to ordinary code. A planner returns a list of steps, a classifier returns a label, a reviewer returns a verdict with a reason, and a tool call carries arguments that a real function is about to run with. For a long time the standard approach was to ask nicely in the prompt, parse whatever came back, and retry when it broke. All three major model providers now sell a better option: you send a JSON Schema, and the model's output is constrained to match it. This post looks at what that guarantee really covers on each platform, the parts of your schema that are silently dropped, and the handful of cases where you still get something that does not match.

Two features with one name

It helps to separate two things that the documentation often discusses together. The first is shaping the model's final answer. The second is shaping the arguments of a tool call. Anthropic's structured outputs page is explicit about the split: JSON outputs, sent in output_config.format, control "what Claude says," while strict tool use, turned on with strict: true on a tool definition, controls "how Claude calls your functions." You can use either one alone or both in the same request, which the page recommends for agentic workflows that need reliable tool calls and a structured final result.

OpenAI's Structured Outputs guide draws the same line. Structured Outputs is available through function calling and through a json_schema response format, and the guide's advice is to use function calling when the model needs to reach into your application and the response format when you want the model's reply to the user in a fixed shape. It also separates the feature from the older JSON mode: "While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence." If you still have response_format: {type: "json_object"} somewhere in an agent, that call is only promising syntax, not structure.

Google's Gemini structured output documentation covers the response side, configured by setting the response MIME type to application/json and supplying a schema. One detail matters for agents: combining structured output with built-in tools and function calling in the same request is marked as a preview feature available only to Gemini 3 series models. If your agent needs both a grounded search and a typed final answer in one call, check which model you are on before designing around it.

How the guarantee works, and what it costs

The mechanism is constrained decoding. Anthropic's page describes compiling your schema into a grammar and restricting sampling to tokens that keep the output valid, which is why it can promise "No more JSON.parse() errors" and "No retries needed for schema violations." That compilation is not free. On Anthropic, the first request with a new schema pays extra latency while the grammar compiles, and compiled grammars are cached for 24 hours from last use. Changing the schema structure, or the set of tools when you use both features, invalidates that cache, while changing only names or descriptions does not. Anthropic also injects an extra system prompt explaining the expected format, so input tokens go up slightly, and changing output_config.format invalidates the prompt cache for that conversation. OpenAI's guide notes a first-request delay only for fine-tuned models and says other models do not have it.

For agent builders the practical lesson is to treat schemas like code that gets deployed. If a harness builds a slightly different schema on every turn, for example by inserting the current user's name into an enum or reordering tools at random, it can keep paying compilation and cache costs that a stable schema would avoid. We made a similar point about stable prefixes in our guide to tool-calling costs.

The schema subset is smaller than JSON Schema

None of the three providers accepts all of JSON Schema, and they fail in different ways when you go outside the subset. That difference is the most common source of surprise.

OpenAI rejects what it does not support. Its guide says every field must be listed in required, and that you emulate an optional field with a union type that includes null, such as "type": ["string", "null"]. Every object needs additionalProperties: false. A schema may have up to 5,000 object properties and 10 levels of nesting, up to 1,000 enum values across all enum properties, and a combined length of 120,000 characters for property names, definition names, enum values and const values. Numbers can use minimum, maximum and multipleOf, and arrays can use minItems and maxItems. Composition keywords such as allOf, not, if, then and else are not supported. On the plus side, OpenAI produces keys in the same order as your schema.

Anthropic also returns a 400 error for unsupported features, but its SDK helpers will quietly rewrite your schema first. According to its documentation, the helpers remove constraints such as minimum, maximum, minLength and maxLength, copy them into the field's description so the model still sees the intent, add additionalProperties: false to every object, and then validate the response against your original schema. Recursive schemas, numeric and string length constraints, and array constraints beyond a minItems of 0 or 1 are not supported. Anthropic also publishes explicit complexity limits: at most 20 strict tools per request, 24 optional parameters in total across all strict schemas, and 16 parameters that use anyOf or type arrays. Those totals are summed across the whole request, so four strict tools with six optional parameters each already hit the 24 limit. Beyond them, the API can still answer "Schema is too complex for compilation," and it enforces a 180-second compilation timeout.

Gemini takes a third approach. Its documentation lists minLength, maxLength, minProperties, maxProperties, uniqueItems, contains, minContains and maxContains as constraints the decoding engine does not enforce, and says the API "automatically strips and ignores them during request normalization rather than returning an error." It specifically warns that Pydantic's Field(min_length=...) and Zod's .min() emit these keywords for you. minItems and maxItems are enforced. Very large or deeply nested schemas can be rejected with a 400 INVALID_ARGUMENT error about too many states, and the page lists large maxItems limits on nested arrays, long property or enum names, big enums, long runs of optional properties and wide anyOf unions as the usual causes.

Here is a concrete example of how that bites. Say a summarizer step in your agent uses a Pydantic model with summary: str = Field(max_length=280) because the output is posted somewhere with a character limit. On OpenAI, the guide's list of supported string properties includes pattern and format but not length limits, and it says that calling the API with strict: true and an unsupported schema returns an error, so you are likely to find out on the first request rather than in production. On Anthropic, the SDK helper moves the limit into the description and validates afterward, so an overlong summary becomes a validation exception in your code. On Gemini, the keyword is stripped and the call succeeds, so an overlong summary simply arrives. Same schema, three behaviors. The fix is the same everywhere: keep your own validation step after parsing, which Gemini's page recommends directly with Pydantic's model_validate_json() or Zod's .parse().

When a valid response still does not match

Constrained decoding does not cover every path out of the model, and each provider documents the exceptions.

Refusals are the first. OpenAI says that because a refusal does not necessarily follow your schema, the response carries a separate refusal field, and its examples check for a refusal content item before reading the text. Anthropic returns a normal 200 status with stop_reason: "refusal", bills the generated tokens, and warns that the output might not match your schema. It adds a subtle one: a schema property that asks for the model's step-by-step reasoning can itself trigger a refusal, and the page suggests asking for a short explanation instead.

Truncation is the second. If the model hits its output limit, the JSON can be cut off mid-object. Anthropic signals this with stop_reason: "max_tokens" and suggests retrying with a higher limit. OpenAI's example checks for status === "incomplete" with the reason max_output_tokens. An agent loop that parses blindly will turn a truncated answer into a crash, or worse, into a confusing error message the model then tries to reason about. Our post on handling tool errors in agent loops covers how to feed failures like this back cleanly.

The third is quieter. Anthropic documents that string enum and const values are not guaranteed to keep their capitalization, typically for the first letter of a word after a space. An enum of "Conversation topic 3" can come back as "Conversation Topic 3", with no error and no special stop reason. Its advice is to compare enum values case-insensitively and avoid values that differ only by case. Anthropic also notes that required properties are emitted before optional ones, so if your parser depends on key order, either mark every property as required or stop depending on order.

Finally, every provider is careful to say the guarantee is about form, not truth. Gemini's page puts it plainly: output is "syntactically correct JSON," but you should still validate values and handle "schema-compliant but semantically incorrect outputs." A perfectly typed tool call can still point at the wrong customer ID.

A pattern that holds up

Putting the documentation together, a sturdy setup looks the same on every provider. Keep schemas small, flat and stable, with every field required and nullable where you mean optional, because that shape is accepted by OpenAI's rules, keeps Anthropic's optional-parameter count low, and avoids Gemini's branching limits. Reserve strict mode for the tools where a malformed call does real damage, which is also Anthropic's first tip for staying under its complexity limits. Check the stop reason or status before parsing, and route refusals and truncations to their own handling. Then validate the parsed object with the same Pydantic or Zod model you used to generate the schema, so constraints the provider dropped are still enforced somewhere. If you write evaluations for your agent, add a few cases that deliberately produce overlong strings or near-miss enum values; our guide to evaluating agent tool calls shows how to fold those into a regular suite.

Structured outputs remove a whole class of parsing bugs, and they are worth turning on. They just move the remaining checks to a smaller, more predictable place, and knowing exactly where that place is on your provider is what keeps an agent from failing quietly.

For a broader walkthrough of embeddings, retrieval and modern LLM tooling that pairs well with schema-first agent steps, Hands-On Large Language Models by Jay Alammar and Maarten Grootendorst is a clear practical companion.

If you run models on your own machine, we keep a short list of local LLM gear — GPUs, Macs, RAM and storage — with plain-text Amazon links.