Hero image for "The Advice That Worked Is a Biased Sample"

The Advice That Worked Is a Biased Sample


Here's a pattern worth sitting with. You read a book about how a founder built a billion-dollar company. You read another about a different founder who did the same. You pull out the common threads — the relentless focus, the contrarian conviction, the willingness to ignore early critics — and you treat those threads as lessons.

What you've actually built is a model of survivorship.

The founders who failed also had relentless focus. They also had contrarian conviction. They also ignored early critics. The difference between the successes and the failures often has less to do with the traits the books identify and more to do with things that are harder to narrativize: timing, market dynamics, a competitor's stumble, a single fortuitous introduction. But those factors don't make for clean frameworks, so they get compressed into footnotes while the hero traits get the chapters.

This is survivorship bias in its most seductive form — and it runs through startup advice like a structural flaw.

The Graveyard Doesn't File Postmortems

Britannica defines survivorship bias as a logical error where focus is placed only on things that have survived some process, overlooking those that did not — with successes receiving more attention than failures, intentionally or unintentionally suppressing some evidence to highlight other evidence.

The classic illustration is Abraham Wald's WWII analysis of returning bombers. Engineers mapped bullet holes on the planes that came back and recommended reinforcing the damaged areas. Wald's insight, documented at Bytepith, was the opposite: the returning planes had already demonstrated they could absorb hits in those spots and survive. The planes hit in the undamaged areas never made it home to be counted. Reinforce the places with no holes.

The startup advice ecosystem has exactly this problem. The companies that could speak publicly about their lessons are the ones still around to speak. The ones that died took their actual lessons with them — or got compressed into a brief TechCrunch postmortem that nobody studies the way people study Steve Jobs biographies.

IdeaProof's analysis of over 1,000 startup post-mortems puts the base rate starkly: roughly 90% of startups fail, and the single most common root cause — accounting for about 42% of failures — is no validated market need. Not bad execution. Not weak teams. The market simply wasn't there. Meanwhile, the success stories we study are disproportionately populated by companies that happened to find real demand, often after a pivot, often through luck of timing.

What the "Excellence" Playbooks Actually Proved

The mechanism shows up most clearly when someone actually tests it.

In 1982, Tom Peters and Robert Waterman published In Search of Excellence, profiling companies they judged excellent and extracting eight shared traits meant to explain their performance. Five years later, as Bytepith reports, analyst Michelle Clayman did something the original authors hadn't: she built a comparison group. She tracked the "excellent" companies against a portfolio of companies scoring worst on the same financial criteria.

The unexcellent portfolio beat the S&P 500 by more than 12% a year. The excellent companies beat it by roughly 1%. The traits meant to explain excellence showed no advantage once a non-survivor-filtered comparison was actually tested. The book had been studying the planes that came back.

The same distortion runs through investment advice. Researchers who rebuilt the full population of U.S. domestic equity mutual funds from 1993 through 2006 — adding back every fund that had closed or merged out of existence — found an average annual alpha of -0.95% against the broader market. Restrict the same calculation to funds still open at the end of the period, the version investors actually see in marketing materials, and the number flips to +0.14%. The sample quietly dropped its failures, and the conclusion reversed.

What Good Pattern-Matching Actually Requires

None of this means startup case studies are useless. It means you need to hold them differently.

The lesson from IdeaProof's postmortem database isn't just that 42% of startups die for lack of market need — it's that expensive failures and expensive successes often look identical from the inside in months one through eighteen. The food delivery company described by one founder in Medium's Write A Catalyst was earning $2 million a year while losing $4.20 on every order — real revenue, fabricated unit economics, the appearance of traction and the reality of a slow bleed.

The advice you get from winners is real advice about what those specific people did in those specific conditions. What it can't tell you is how many other people did the same things in similar conditions and failed anyway. That's the missing data. That's the graveyard.

The discipline is asking, for any piece of startup wisdom: what would have to be true for this to be noise? If someone built a company with relentless focus and won, how many founders applied relentless focus in the same year and lost? The traits the winners share are also the traits the losers share — we just stopped collecting data on the losers.

When you read the next framework that promises to explain success, look for the comparison group. If it isn't there, you're reading a book about planes with bullet holes — studying the damage that didn't kill anyone, and drawing conclusions about what keeps planes in the air.