Hero image for "Your Canary Is Comparing the Wrong Model to the Wrong Model"

Your Canary Is Comparing the Wrong Model to the Wrong Model


A support team rolled out a new refund-approval model. It beat the old one on five hundred historical conversations — 91% right versus 87% — so they shipped it to everyone on a Tuesday. Latency held at 410ms. Error rate stayed at zero. Every dashboard was green. A week later, someone pulled the actual refund outcomes and found the new model had approved a wrong refund one in every eight times, up from one in forty the month before (DEV Community).

Here's the detail that should bother you more than the error rate: nobody could say which model version had actually answered any of those conversations. Every log line read "model":"latest" (DEV Community). latest isn't a version — it's a pointer that moves whenever the vendor decides it moves. The team's config asked for latest, so Tuesday's traffic silently got a new model nobody chose to deploy that day.

I wrote last month about canaries that route traffic but never judge it — a canary that splits requests without scoring the output is just a percentage, not a safety net (/p/a-canary-that-never-gets-judged-is-just-a-traffic-split). This is the layer underneath that problem. Before you can judge whether a canary is healthy, you have to know with certainty which model produced each answer. A surprising number of teams can't.

Your logs record what you asked for, not what you got

The refund case is a logging gap. A version-aliasing GitHub incident shows what that gap costs you during an actual investigation. A judge pipeline went silent for a month — every LLM call failed upstream and silently fell back to a deterministic keyword matcher, which almost never cleared the urgency threshold the system needed (GitHub). Two separate diagnosis attempts during that month pinned the blame on a model swap, because the cost ledger field everyone checked recorded the requested model, not the one that actually succeeded. Both diagnoses were wrong and had to be withdrawn once someone noticed the ledger column was lying by design, not by bug.

That's the pattern underneath both incidents: the field that looks like "which model handled this" is actually "which model we asked for." Those are the same value right up until something fails upstream — a provider outage, a silent fallback, a moving alias — and then they diverge in exactly the window where you need them not to. Pin the exact version in config, and log the version the response actually claims it answered with, not the one in your request (DEV Community). If your provider's API returns a resolved version string, that field exists specifically so you can stop guessing.

A passing eval score is a statement about last month

The refund team's 91%-versus-87% comparison wasn't wrong, exactly — it was stale the moment they shipped it. A model is graded against real conversations, and real conversations keep changing in ways a code test suite never does. A score on last month's traffic tells you how the candidate would have done last month (DEV Community). TechTarget's framing of model-change management makes the same point from the vendor-dependency side: version changes and deprecations arrive on the provider's schedule, not yours, and treating a minor version bump as low-risk just because the API shape didn't change is how teams get surprised (TechTarget).

The operational fix isn't exotic. A canary-evaluation-gate pattern worth copying freezes the whole release as a bundle — prompt template, model config, tool schemas, retrieval config, and evaluator version all resolve to one identifier, with the rendered prompt hash recorded alongside the template version, because dynamic assembly can change behavior without touching the template itself (OneUptime). Shadow deployment helps here too — running the candidate on a copy of real traffic without ever surfacing its output lets you diff predictions against the incumbent at zero user risk, though it still can't tell you how outcomes shift once users actually see the new answers (Opsware). And when something does go wrong, a postmortem template with a mandatory field for prompt ID, model version, and tool release tag turns "we think it was the new model" into something you can actually verify months later (DEV Community).

None of this is complicated. It's bookkeeping. But it's the bookkeeping that decides whether your next incident review spends a day confirming facts or a month retracting two wrong diagnoses because the one field everyone trusted was recording the wrong question.