You've got two prompt variants. You run them against a traffic split. Variant B wins on your quality metric. You ship it.
Three days later, something unrelated starts degrading — a summarization step, a classification branch, a downstream formatter. You dig in and realize: the "winning" prompt shared a base policy component with six other prompts, and the change you made to win the A/B test quietly altered behavior across all of them.
This is the part of prompt experimentation that nobody talks about at the stage where it actually hurts teams.
The Dependency Problem Arrives Before the Testing Problem
Most teams reach for A/B testing infrastructure when they feel the pain of not knowing which prompt variant is better. That's the right instinct. But the harder problem — the one that bites you in production — is understanding what a prompt change touches before you test it.
A recent post on Towards Data Science describes building a prompt dependency graph on a 55-node synthetic system to answer exactly this question. The graph separates two things: the reachable set (everything structurally downstream of a changed component) and the candidate set (what actually needs targeted evaluation because it directly depends on the changed section). Depending on how selectively a component is shared, section-aware tracking narrowed the evaluation set by anywhere from 0% to 85% across different change scenarios.
The uncomfortable finding: you don't know which outcome you're getting until you run it. Some changes look locally contained and cascade everywhere. Others look broad and turn out to be isolated. The graph doesn't tell you what will fail — it tells you what you need to evaluate. That's a different, more honest answer than most teams are getting right now.
If your prompts are composed from shared pieces — base policies, tone instructions, format templates — and you're running A/B tests without this kind of dependency awareness, you're not testing variants. You're testing the visible surface of a change while the downstream effects run unmonitored.
What "Production-Grade" Prompt Management Actually Requires
The operational gap here is well-documented. As cloudrps.com describes the LLMOps problem: the "model" in a production LLM system is actually a combination of the base model version, the system prompt, few-shot examples, retrieval configuration, temperature settings, and post-processing logic — and almost none of these get versioned consistently in teams that haven't explicitly built LLMOps practices. Every one of those components affects output quality. All of them change independently.
This is why prompt A/B testing at production volume requires more than a traffic splitter and a quality metric. You need:
- Version control with diff visibility — not just "which version is running" but "what exactly changed between v4 and v5 of this component"
- Dependency tracking — knowing which downstream prompts consume a shared component before you promote a variant
- Rollback that actually works — meaning instant revert to a known-good state, not a redeploy cycle
LaunchDarkly's enterprise prompt management tooling frames the problem clearly: without runtime control, prompts live in config files, Slack threads, and spreadsheets, and nobody knows which version is running in production. Every prompt change requires an engineering PR and a full deploy cycle, which means product can't iterate and engineering becomes the bottleneck. The solution they're selling is runtime prompt updates with full version history, side-by-side diff comparison, and A/B testing across cohorts — but the underlying problem description is accurate regardless of vendor.
The Metric You're Optimizing Might Be the Wrong One
Even when teams have the infrastructure right, the evaluation signal is often wrong. Big Data Boutique's production AI engineering overview puts it plainly: LLM evaluation requires human judgment or another LLM acting as judge, you can't write unit tests that assert exact string equality, and the signal is noisier and more expensive to run at scale than traditional software testing.
This matters for A/B testing specifically because the metric you pick to declare a winner tends to be whatever's easiest to measure — token count, latency, a simple relevance score. The harder quality signals (does this response actually serve the user's intent? does it hold up across edge cases?) require LLM observability tooling that traces the full request lifecycle and surfaces evaluation scores alongside cost and latency data. Without that, you're optimizing for a proxy metric and calling it a win.
The teams that get this right treat prompt experimentation as a continuous loop, not a one-time comparison. They run experiments, measure real metrics across cohorts, monitor for downstream degradation, and maintain the dependency graph that tells them what to retest when something changes.
The Discipline That Scales
The dependency graph approach from the Towards Data Science post is worth building even if you never run a formal A/B test. Just knowing the reachable set for any given shared component — before you touch it — changes how carefully you approach changes. It forces you to treat prompts as a system rather than a collection of independent strings.
At production volume, that's the only frame that holds. Prompt engineering stops being about crafting better instructions and becomes about managing a distributed configuration system where every component has consumers you need to account for. The teams that scale prompt experimentation successfully are the ones who built that operational discipline first, then added the testing infrastructure on top of it.
