Pick any major AI model release in the past year. Somewhere in the announcement, you'll find a table: rows of benchmark names, columns of model versions, and a satisfying diagonal of bold numbers showing the new model winning. What that table almost never shows is what the benchmarks were actually measuring, whether the training data touched the test set, or what happened on the evaluations that didn't make the cut.
This is the core problem a new preprint on arXiv takes on directly — and it's worth reading carefully, because it names the mechanism rather than just the symptom.
The Proxy-Target Gap Has a Name Now
The paper, EvalSafetyGap, is an 80-page hybrid survey and conceptual framework synthesizing 373 studies published between 2018 and 2026. Its central argument is that benchmark scores, reward signals, and safety metrics can all improve while the underlying capabilities and alignment properties they're supposed to measure remain uncertain or actively deteriorate. The authors call this "proxy-target divergence under optimization pressure" — a formal way of saying that when a measure becomes a target, it stops being a good measure.
That's Goodhart's Law, which economists have understood for decades. What's new here is the application to AI safety specifically, and the taxonomy of how it breaks down. The paper identifies benchmark saturation, data contamination, prompt sensitivity, and construct-validity failures as distinct failure modes — not interchangeable symptoms of the same vague problem. A model that scores well because its training data overlapped with the test set is failing differently than a model that scores well because the benchmark was never a valid proxy for the capability in question. Both failures look identical in the results table.
The framework introduces what the authors call an Alignment Trilemma: capability, behavioral robustness, and governance disclosure are three separate evidence layers that current evaluation practice tends to collapse into a single safety score. An exploratory audit of ten publicly available models illustrates why that collapse is misleading — the paper argues these dimensions should be reported separately rather than aggregated, because a model can score well on one while failing badly on another.
Cherry-Picking Has Infrastructure
The more uncomfortable implication isn't that researchers are lying. It's that the system is structured to produce optimistic-looking results without anyone making a deliberate choice to deceive.
Consider how benchmarks get selected for publication. A lab runs evaluations across many tasks. Some results are strong; some aren't. The paper goes out with the strong ones prominently featured. This isn't fabrication — it's selection, and it's nearly universal. The EvalSafetyGap survey documents this pattern across the literature: benchmark validity weaknesses and data contamination are recurring findings, not edge cases. The paper specifically flags that "model-mediated evaluation" — using one LLM to judge another — introduces additional distortions, since the judge model may share the same training distribution as the model being evaluated.
The adversarial safety testing literature adds another layer. Jailbreaks and refusal failures documented in research settings often don't appear in the benchmark suites labs use to report safety performance. The paper describes these as "potentially brittle safety mechanisms" — they hold under the conditions tested and may not hold under conditions that weren't. The research agenda the authors propose includes pre-specified multi-attempt threat models, which would require labs to test against adversarial conditions before claiming robustness rather than after a failure surfaces publicly.
What "Transparent" Actually Requires
The paper's proposed remedies are methodologically specific in a way that's useful. Version-locked evaluation — running the same benchmark suite against a fixed model version rather than allowing iterative updates to the model between evaluation and publication — would make it harder to quietly improve scores before reporting. Transparent source reporting would require labs to disclose which data sources were used in training, enabling independent contamination checks. Dynamic and contamination-resistant benchmarks would rotate test items to prevent training-set overlap from accumulating over time.
None of these are technically difficult. They're institutionally difficult, which is a different problem. Labs have competitive incentives to report strong numbers, and the current publication ecosystem doesn't penalize selective reporting the way it would penalize outright fabrication. The EvalSafetyGap framework is explicit that this is a governance problem as much as a measurement problem — the research agenda it closes with includes AI auditors as a distinct stakeholder class who need a "shared vocabulary for measurement-aware LLM safety evaluation."
That vocabulary is what's been missing. When Nature recently covered the release of Kimi K3 — Moonshot AI's new reasoning model — the coverage noted that "the company's own tests found that K3 can match or outperform rival US models." The phrase "company's own tests" is doing a lot of work there. It describes a benchmark claim while quietly flagging that the claim comes from the party with the most to gain from it. That qualifier appears in responsible science journalism. It rarely appears in the benchmark tables themselves.
The same dynamic plays out at the infrastructure level. Google's Q2 2026 financials show the company planning to spend as much as $205 billion on AI infrastructure this year — capital commitments of that scale create enormous pressure to demonstrate that the underlying models are performing and safe. When the investment thesis depends on capability claims, the incentive to report the most favorable benchmark results isn't a bug in the system. It's load-bearing.
Watch for whether any of the major labs adopt version-locked evaluation protocols in their next round of model releases. That's the concrete test of whether this framework moves from survey literature into actual practice.
