A RAG-based customer support assistant shipped to hundreds of users before anyone noticed it was confidently citing billing policies that didn't exist. The post-mortem finding, per one team's retrospective: zero automated evaluation. Their process was literally "ask questions, read answers, thumbs up."
That's not a testing gap. That's no testing at all. And it's more common than anyone wants to admit.
The problem is that most small teams treat LLM evaluation as a pre-ship ritual — run some prompts manually, feel okay about it, deploy. What they're missing is that prompt changes, model upgrades, and retrieval strategy tweaks all introduce regressions that manual spot-checks won't catch. You need evaluation wired into your deployment pipeline the same way you'd wire in unit tests. Not because it's theoretically correct, but because the failure mode for skipping it is a hallucinating assistant reaching real users at scale.
The Three-Layer Stack You're Probably Missing
A well-structured eval pipeline has three components that most teams implement partially or out of order.
Dataset management first. Your eval dataset is your test suite, and it needs to be version-controlled like code. That means golden examples curated by domain experts, known edge cases and adversarial inputs, and — critically — real traffic samples from production logs. The last category is what most teams skip. Synthetic test cases and hand-crafted examples will miss the weird, ambiguous, domain-specific inputs your actual users send. If your eval dataset doesn't include logged production inputs, you're testing a cleaner version of your product than the one people are actually using.
Metrics that match your use case. Reference-based metrics (exact match, ROUGE scores, semantic similarity) work when you have ground truth. For open-ended tasks — diagnosis, summarization, customer support — you need reference-free metrics or LLM-as-judge. The industry has largely converged on LLM-as-judge for complex tasks: you use a separate, capable model to score outputs against a rubric you define. The practical implementation scores on dimensions like accuracy, completeness, clarity, and actionability — each on a 1-5 scale — and returns structured JSON you can track over time. The threshold you set (what score counts as "pass") is a product decision, not a technical one. Own it explicitly.
CI/CD integration as the actual gate. This is where most teams stop short. They build evaluation tooling but run it manually before big releases. The pattern that actually catches regressions is triggering evals on every pull request that touches prompts or eval configs, plus nightly runs against production traffic samples. When a PR drops your hallucination score below threshold, it fails the build. Prompt changes are code changes — I've written about this before — and the eval pipeline is what makes that principle operational rather than aspirational.
The Hidden Cost of LLM-as-Judge
One thing vendor documentation glosses over: running a capable judge model against every eval case costs money, and at scale it adds up fast. If you're running nightly evals across a few hundred test cases using a frontier model as judge, you're paying for that compute. I'd argue this is worth it — catching a hallucination regression before it ships is cheaper than the trust repair afterward — but you need to budget for it explicitly.
The practical mitigation is tiered evaluation. Fast, cheap metrics (format checks, length constraints, keyword presence) run on every PR. LLM-as-judge runs on a representative sample, or only when fast metrics flag something. Full eval suite runs nightly. You're not skipping coverage — you're sequencing it so the expensive checks run where they matter most.
The other cost that doesn't show up in your inference bill: eval dataset maintenance. Your test cases need to evolve alongside your application. Previously reported production issues should feed back into the dataset. If you shipped a hallucination and fixed it, that failure case belongs in your golden set permanently. Treat it like a regression test in traditional software — the bug you fixed is the test you keep.
What to Actually Watch
The teams that get this right aren't running more evals — they're running evals on the right triggers. Two concrete things to wire up if you haven't:
Path-based CI triggers on prompts/** and eval/** directories. Any change to prompt files or eval configs kicks the pipeline. This catches the "quick prompt tweak" that nobody thought to test.
Score trending over time, not just pass/fail on individual runs. A model that scores 4.1 today and 3.8 next week and 3.4 the week after is failing in slow motion. Point-in-time pass/fail won't catch it. Tracking score distributions across nightly runs will.
The eval pipeline is infrastructure. It has maintenance overhead, it has compute costs, and it needs to be owned by someone. The alternative is finding out your model regressed from a customer complaint instead of a failing build.
