Hand the same dataset to a dozen competent researchers and ask them to test the same hypothesis, and you'll get a dozen different answers — not because anyone cheated, but because "the same analysis" is rarely as well-defined as it sounds. That's the premise behind a wave of recent work testing what happens when you formalize the disagreement instead of treating it as noise to be embarrassed about.
Two new projects, one published in PNAS and one from the International Institute for Replication, take this problem seriously enough to actually measure it at scale — and what they find complicates the standard "many analysts" horror story in useful ways.
The Privacy Noise Scare Turns Out to Be Smaller Than the Noise Already Baked Into Social Science
Start with the most direct test of robustness I've seen in a while. A team replicated 93 published social science studies, all using regression analysis on aggregate statistics, and asked a specific question: if you inject the kind of statistical noise that agencies like the US Census Bureau now add to protect privacy, do the original findings survive? The result, reported in PNAS, is that under industry-typical privacy budgets, about 91% of the simulated findings still supported the original claims at a 0.1 significance threshold. More striking: the distortions from privacy noise were consistently smaller than the disagreements that show up in ordinary replication and robustness studies — the kind that have nothing to do with privacy at all.
That's a useful corrective to a loud debate. Critics of differential privacy have argued that injected noise would quietly wreck social science's evidence base. The data say the opposite: the "many analysts" variance that already exists in how researchers specify models, choose controls, and define samples dwarfs the variance introduced by a privacy-preserving algorithm. The scarier source of disagreement was never the noise added on purpose. It's the noise baked into ordinary analytical choice.
When You Pay People to Disagree Carefully, the Disagreements Get More Informative
The second project goes further: instead of asking what happens when noise gets added to data, it asks what happens when you deliberately hire independent people to stress-test the same findings under a tightly specified protocol. The I4R's R2E-Dev initiative, described in an I4R blog post, recruited paid replication fellows — compensated 2,500 EUR each — to audit code and run pre-specified robustness checks on 66 empirical studies drawn from top development economics journals. The project's organizing principle is deliberately conservative: robustness checks have to stay close to the original research question and be statistically defensible, rather than wandering into "what if we'd measured this differently entirely" territory. The goal, as the authors put it, is to establish a higher bound for how robust published economics findings actually are — a ceiling, not an indictment.
What makes this interesting methodologically is the standardization. Most "many analysts" studies — crowdsourced efforts where dozens of teams independently analyze one dataset — generate a scatter of results and a headline about how wildly scientists disagree. R2E-Dev inverts that: it pins down the analytical latitude first, then measures how much conclusions move within those guardrails. That's a narrower question than "how much do scientists disagree," but it's a more answerable one. It tells you how much of the disagreement in published economics work is attributable to genuinely contestable analytical choices versus how much would show up even among researchers trying hard to do the same thing the same way.
The Real Lesson Is About Which Variance You're Measuring
Put these together and a pattern emerges that's more useful than "scientists can't agree." The variance in conclusions from the same dataset comes from at least three distinguishable sources: measurement error and data quality, deliberate methodological choices (model specification, sample restrictions, outlier handling), and — per the PNAS study — engineered noise added for reasons like privacy. These sources are not the same size, and conflating them is how you end up with headlines implying that any analytical disagreement is a crisis. The R2E-Dev framework, by standardizing what counts as a "fair" robustness check before running it, is one of the more rigorous attempts I've seen to separate defensible disagreement from the kind that should worry us.
The R2E-Dev reports are being released gradually, and the full meta-replication results aren't out yet. When they land, the number worth watching isn't how many of the 66 studies "failed" — it's how the Robustness Dashboard the team built actually partitions the disagreement. If it can show which kinds of analytical choices move effect sizes and which don't, that's a tool worth more than another study insisting science is broken.