Survivorship bias
The first time someone explained survivorship bias to me, I remember being a little stunned — the kind of idea that quietly rearranges things you thought you already understood.
The usual example is from the Second World War. Engineers looked at the bombers coming back from missions, mapped where the bullet holes clustered, and wanted to reinforce those spots. A statistician stopped them. The holes were on the planes that made it home. The places with no holes were exactly where you couldn’t afford to be hit — because the planes hit there never came back to be counted.
You get into the data by surviving. So the data is quietly lying to you about what actually matters.
The “proven track record” trap
I think about survivorship bias every time a PM job ad asks for “a proven track record of successful launches” or “demonstrated success in delivering high-quality products.” On the surface, it sounds reasonable.
But there is survivorship bias in product management hiring: companies study the planes that came back. They look for people attached to visible product wins — while ignoring how much of “success” depends on timing, budget, leadership support, market context, engineering capacity, and luck.
We build an entire idea of what a good product manager looks like out of the ones whose products flew. We read their threads, copy their frameworks, treat their playbook as the cause of the outcome.
Meanwhile, the product managers who did equally sharp work — clear thinking, right calls, hard trade-offs — on a product that died for reasons well outside their control (the market wasn’t there, the timing was wrong, the money ran out) simply never show up in the sample.
Same judgment. No medal. No thread. The planes that didn’t come back.
Let me be careful here, because this argument has an ugly twin. I am not saying outcomes don’t matter, and I’m not saying results are unfair so let’s wave them away. Outcomes matter a lot. What I’m saying is more delicate: a single outcome is a noisy signal of what the product manager actually contributed. The result you can see is skill × market × timing × funding × luck — and you cannot cleanly pull the person’s part back out of the finished number.
An example from Android
Somewhere in the middle of building my GymBuddy app, I came across a recommendation to read “Androids: The Team That Built the Android Operating System” by Chet Haase. GymBuddy is an Android app, so I got curious. Today Android feels so natural that you may never have wondered how it was created.
What struck me is that the book names failure as a cornerstone of Android’s success. Companies like Be, WebTV, and Danger — most of which never reached mass-market dominance — are described as a “farm team” for Android. Out of the first hundred employees, nearly everyone had built similar systems before and was now working on things they had already made mistakes on. Dianne Hackborn, for example, used the failure of Palm OS 6 to make sure Android could scale across screen densities and use cases from the start.
Steve Horowitz, who managed the engineering team, put it directly: in tech, “you learn probably more from the failures than you do from the successes.”
And the team didn’t treat their success as inevitable, either. Almost every specialist interviewed in the book attributes the result to being in the right place at the right time — without the iPhone launch scaring the industry and driving manufacturers into Google’s arms, Android might have been another failed footnote. They don’t credit pure genius. They describe readiness meeting an opening in the market that they didn’t create but were prepared for.
Skill made them ready. Timing made them winners. Those are two different things — and a hiring filter that only counts wins can’t tell them apart.
What I take from this
Now, when I evaluate a vacancy and a client, I sympathize more with those who ask not “what did you ship,” but: what did you get right, and how did you know? Where did the real complexity live? What would you do again, and what did you just get lucky on?
An outcome happened once — in one market, with one team, under conditions that won’t repeat. Reasoning travels. It’s the part of a person that shows up again on the next product, whether or not the last one flew.
So when you’re hiring a PM, “tell me about a hard call you made — and how you’d know today if it was the right one” is a better question than “tell me about a product you launched,” even if it’s harder to answer.
Leave a Reply