A model returns syntactically perfect JSON. It matches your schema field for field, types checked, required keys present. And it's still wrong — an enum value in the wrong case, a confidence score of 0.95 attached to a hallucinated fact, a field the model filled in because your schema gave it no way to say "I don't know." Your validator passes it straight through to the database.
That's the trap teams fall into once they "solve" structured output. One engineer describes the pre-schema baseline bluntly: an LLM pipeline that "returns valid JSON eighty percent of the time and something subtly wrong the other twenty percent, with zero indication in the logs that anything broke" (cloudrps.com). Schema enforcement fixes the first half of that sentence. It does almost nothing for the second.
The Guarantee Is Narrower Than It Sounds
Constrained decoding — strict JSON mode, grammar-based generation, strict tool calling, whatever a given provider calls it — works by zeroing out the probability of any token that would break the schema at every decoding step (howaiworks.dev). That's a real guarantee and it's worth having: a form with checkboxes instead of a blank sheet, in the source's own framing. You cannot write outside the allowed options.
But you can still check the wrong box. The guarantee covers syntax, not content — values still need validation in code, and a tight enum can push the model into the nearest wrong option when none of the choices actually fit (howaiworks.dev). I'd put it more plainly for a production audience: schema enforcement is a compiler, not a code reviewer. It tells you the output will parse. It says nothing about whether the parsed value is true.
The reference guide at llmbestpractices.com lays out exactly where schema compliance still fails even in strict mode: a stop_reason of refusal or max_tokens can hand you a truncated or non-conforming response that needs to be caught before you even try to parse it, and enum values can come back in different capitalization than you specified, which means naive equality checks silently drop valid matches (llmbestpractices.com). If your validator does a case-sensitive == against an enum and treats the mismatch as a schema failure, you've turned a provider quirk into a false-positive outage.
Three Places Validation Still Needs to Do Real Work
Strip away the marketing language around "guaranteed JSON" and three concrete jobs remain for your own code:
- Catch the escape hatches. Check
stop_reason/finish_reasonbefore you even attempt a parse. A response cut off by a token limit or blocked by a refusal isn't a schema failure, it's a different failure mode entirely, and conflating the two makes your error logs useless (llmbestpractices.com). - Normalize before you compare. Enum and literal fields need case-insensitive comparison as a baseline, not an edge case (howaiworks.dev).
- Never regex your way around a missing schema mode. If a provider or self-hosted model doesn't support constrained decoding for a given call, parsing JSON out of prose with a regex over
{...}breaks on nested braces, code fences, and multiple objects in one reply — use a balanced-brace scanner at minimum, and treat it as a stopgap until you can switch to schema-enforced mode (llmbestpractices.com).
MLflow's framing is useful here: pick constrained generation or tool/function calling as your default path, and pair it with runtime validation and an explicit fallback for whatever still slips through (mlflow.org). The fallback is not optional. It's the part of the system that absorbs the cases the grammar can't.
Libraries like Instructor and Outlines exist precisely to wire validation and retry logic around the raw decoding guarantee rather than leaving teams to hand-roll it per endpoint (cloudrps.com), and typed-agent frameworks like PydanticAI push the same discipline into the application layer by making the model's output a real typed object instead of a string your code has to interpret (medium.com).
Treat Schema Mismatches as Infrastructure Failures, Not Model Quirks
The operational shift that actually matters is attitude, not tooling: every schema violation should trip the same alerting path as a failed database write, not get logged as a curiosity and forgotten (cloudrps.com). That only works if the validation failure is visible in your traces in the first place — which is the gap most teams actually have. LLM spans carry prompts, completions, and tool arguments that don't look like anything else in your telemetry stack, and if your pipeline isn't normalizing and routing that data before it leaves the application, you won't see the 2% case where a strict schema still shipped garbage into a downstream table (honeycomb.io).
So the checklist for your next on-call rotation isn't "did we turn on strict mode." It's whether your validator distinguishes a refusal from a malformed response, whether your enum comparisons are case-insensitive, and whether a schema failure lights up the same dashboard a failed deploy would. Strict mode buys you a parser that never throws. It doesn't buy you a system that's telling the truth.
