A RAG-based customer support assistant shipped to production and confidently cited nonexistent billing policies to hundreds of users before anyone caught it. The post-mortem finding, per one engineering team's retrospective: zero automated evaluation. Their entire QA process was "ask questions, read answers, thumbs up."
That's not an edge case. That's most teams.
Hallucination detection in production is where the gap between "we tested it" and "we know it works" becomes expensive. The failure mode is specific: your APM dashboard shows a 200 response in 1.2 seconds, your monitoring tool reports no errors, and somewhere downstream a user just got confidently wrong information that will take three support tickets to unwind. The system worked. The output was garbage.
Here's how to close that gap without burning your token budget doing it.
Eval-Time Checks Don't Survive Contact With Real Traffic
The core problem is that evaluation-time hallucination rates and production hallucination rates diverge — often significantly. Controlled evaluation queries don't cover edge cases, adversarial inputs, or usage patterns you didn't anticipate. Systems that look clean in eval often surface new hallucination patterns within weeks of launch.
This is why the "we ran evals before shipping" defense doesn't hold. Evals are necessary but not sufficient. What you need is a detection layer that runs on actual production traffic, not a curated test set.
The architectural implication, as one production systems breakdown frames it: treat the LLM as a stochastic, untrusted component sitting inside a system you're still responsible for making reliable. Reliability is a property of the system around the model, not the model itself. That reframe changes what you build.
Two Detection Patterns Worth Running in Production
Self-consistency sampling. For a configurable fraction of production queries — typically 2–10% — send the same query twice with temperature above zero and compare the two responses. Flag low-similarity pairs for async review. Critically, this runs in a background evaluation lane and doesn't block the user response. At 5% sampling with one extra query per sampled request, you're looking at roughly 5% overhead on your token bill for those queries. For anything customer-facing where a wrong answer has real consequences, that's a cheap insurance policy.
LLM-as-judge inline scoring. A second model call scores the primary response for hallucination before or after it reaches the user. The practical implementation uses structured output validation — a judge model that returns a score between 0 and 1 with enforced schema, not a free-text verdict you then have to parse. Wire this into CI so prompt changes trigger regression checks automatically. Nightly runs catch drift; PR-triggered runs catch regressions before they ship.
The combination gives you two signals: consistency (does the model agree with itself?) and quality (does a judge think the output is grounded?). Neither is perfect alone. Together they catch different failure classes.
Context Engineering Beats Prompt Patching
When hallucinations do occur, the instinct is to patch the system prompt. Add another constraint. Append "DO NOT invent store locations or phone numbers!" The problem is that after analyzing production execution traces, the model usually isn't ignoring your instructions — it's filling gaps in the context it was given. Prompt warnings are soft constraints. The model treats them as suggestions when it has nothing better to work with.
The more durable fix is architectural: replace soft prompt warnings with enforceable backend contracts. If your agent needs to look up a store location, it should call a tool that returns verified data, not rely on the model to recall it. If a policy exists, it should be retrieved and injected into context, not assumed to be in the model's weights. The practical framing: control the three context entry points — what the model knows at inference time, what tools it can call, and what constraints are enforced by the system rather than requested in the prompt.
This is also where user feedback loops earn their keep. A simple thumbs-down with optional free text, routed to a human review queue, gives you a ground-truth signal that no automated judge can fully replicate. Track negative feedback rate over time as a quality metric. When it ticks up, something changed — a prompt, a retrieval layer, a model version — and you now have a timestamp to investigate against.
The teams that handle this well have stopped asking "did our eval pass?" and started asking "what's our production hallucination rate this week, and what changed?" Those are different questions. The first is a gate. The second is an operational discipline. Build toward the second.
