6 min read
N03

Mind · No. 14

The Planes That Didn't Come Back

The most important evidence is usually the evidence that never made it home.

· · 6 min · 1,469 words

In short

  • During the Second World War, the statistician Abraham Wald pointed out that the damage on returning planes showed where a plane could be hit and survive, not where it was most vulnerable.
  • Survivorship bias happens whenever a filter removes the failures from view and you study only what remains.
  • Most success advice is drawn from survivors, so it often mistakes common traits, or plain luck, for causes.
  • Before trusting any pattern, ask what process produced the sample and what the missing cases would look like.
  • The losers rarely write books, which is why the most important lessons are often the hardest to find.

During the Second World War, a group of mathematicians in New York were asked a practical question with lives attached to it. Bombers were coming back from missions full of bullet holes. Armour would protect them, but armour is heavy, and a plane carrying too much of it flies slower, burns more fuel and becomes an easier target. So where, exactly, should the armour go?

The obvious approach was to look at the planes that returned and count the holes. Some areas were peppered with damage. Others were almost clean. Put the armour where the holes are.

The statistician Abraham Wald saw the flaw. The planes being studied were the ones that had come back. The holes showed where a bomber could be hit and still fly home. The places with no holes were not lucky. They were the places where a hit brought the plane down, so those planes were never in the sample at all.

The armour belonged where the holes weren't.

The man and the story

Wald was born in 1902 in Cluj, then part of Hungary and now in Romania. He was a gifted mathematician who worked in Vienna until the late 1930s, when, as a Jew, he had to flee. He came to the United States and during the war worked with the Statistical Research Group at Columbia University, a small team of exceptional minds doing mathematics for the military.

The popular version of the bomber story, often illustrated with a drawing of a plane covered in red dots, is simplified. Wald's actual work was a set of technical memos on how to estimate the vulnerability of different parts of an aircraft from the damage on the ones that survived. The red-dot plane is a later illustration, not a wartime document. But the core insight is real, and it is one of the most useful ideas anyone has ever written down.

It has a name now: survivorship bias. You study what made it through a filter and forget that the filter removed something.

The places with no holes were not lucky. They were the places where a hit meant the plane never came home.

The filter is everywhere

Once you see survivorship bias, you start finding it in every corner of life.

"They don't build them like they used to." People admire old buildings and assume past builders were more careful. Some were. But the old buildings still standing are the ones good enough or lucky enough to survive. The badly built ones fell down or were torn down long ago, and nobody photographs them for postcards.

"Music was better back then." You remember the songs from past decades that were good enough to still be played. The forgettable ones were, by definition, forgotten. You are comparing the best of the past with everything from the present.

Investment funds. Funds that perform badly tend to close or get merged into other funds. When someone studies "the average fund over twenty years," the failed ones may have quietly disappeared from the data, which makes the whole industry look better than it really was.

Old wisdom about anything. When a grandparent says "we never wore seatbelts and we were fine," they are speaking as a survivor. The people for whom it did not turn out fine are not at the dinner table to disagree.

In each case, the evidence you see has been through a filter. Time, failure or death removed the cases that would have told a different story.

Why most success advice misleads you

Now take this lens to the business section of any bookshop.

Success books follow a familiar recipe. Find a group of successful people or companies. Study what they have in common. Present those traits as the secret. They took big risks. They ignored the doubters. They dropped out of university. They went all in.

The problem is the planes that did not come back.

For every founder who dropped out and built something enormous, there are many who dropped out and did not. For every company that bet everything on one bold idea and won, there are many that made an equally bold bet and vanished. Those people do not get interviewed. They do not write memoirs. Their companies do not become case studies. They are the holes you cannot see.

If you only study winners, you cannot tell whether a trait caused success or was simply common among everyone who tried, winners and losers alike. Risk-taking might be shared equally by people who became billionaires and people who went bankrupt. The difference between them may have been timing, market, help from someone at the right moment, or luck.

Success advice can still be useful, as long as you ask a sharper question. Not "what did the winners do?" but "what did the winners do that the losers did not?"

That second question is much harder to answer, because it requires finding the losers. Which is exactly why so few people bother.

The startup world runs on survivor stories

Startups are an especially dangerous place for this bias, because the outcomes are so lopsided. A tiny number of companies become enormous and generate most of the attention. Thousands quietly shut down and leave little trace.

That shapes what young founders believe. The famous stories of founders who worked out of a garage, ignored everyone and refused to give up are all true. They are also selected. Plenty of founders worked out of a garage, ignored everyone and refused to give up, and the main thing it got them was a longer, more expensive failure.

A more useful lesson than "never give up" goes something like this: notice the difference between persistence on a good idea and persistence on a bad one, and seek out people who failed so you can learn what failure actually looks like from the inside. Their stories are less inspiring. They are often more useful.

The survivors inside your own company

If you build things, the same filter is quietly shaping your decisions every week.

Think about customer feedback. You survey your users, and they tell you what they love and what they want next. But the people answering are the ones who stayed. The people who tried your product, got confused on the second screen and left are not in the survey. Their silence is the most important feedback you will never receive. A team that only listens to current users can keep polishing the parts that already work while the real problem sits exactly where the holes aren't.

The same is true inside a team. The people who stayed for five years can tell you what makes the company great. The people who left in their first three months could tell you what is broken, but they are gone, and nobody asked.

Even your own memory is a survivor. You remember the risky bets that worked and tell them as stories. The ones that failed fade, or get retold as lessons that conveniently prove you were right all along.

How to look for the missing planes

You cannot always find the evidence that disappeared. But you can build habits that make you less likely to be fooled.

  1. Ask what produced the sample. Before trusting any pattern, ask how these examples ended up in front of you. Were they selected by success, by fame, by survival, by someone who wanted to sell you something?
  2. Imagine the missing cases. What would the failures look like? Where would they be? If you cannot picture them, you probably have not accounted for them.
  3. Compare against everyone who tried. A trait only matters if it is more common among winners than among people who attempted the same thing and failed.
  4. Distrust stories that are too clean. Real causes are messy. When an explanation fits neatly into a book title, something has often been left out.
  5. Seek out failure on purpose. Read post-mortems. Talk to founders whose companies closed. Ask experienced people about their worst decisions, not just their best ones.

This kind of thinking is harder than collecting inspiring stories. It is also how you avoid putting your armour in the wrong place.

The silence that matters

What makes Wald's insight so powerful is that it is about absence. Most of our thinking works on what is in front of us. Wald trained himself to ask about what was missing, and to treat that silence as data.

That is a rare habit, and a valuable one. The returning planes did not lie. Every hole was real. The mistake was in assuming the planes that came back were the whole story.

The losers rarely write books. That is why the most important lessons are so hard to find.

So the next time someone tells you the secret to success, look at the holes they are pointing to. Then ask, gently, about the planes that didn't come back.

Further reading

  • How Not to Be Wrong — Jordan Ellenberg
  • Fooled by Randomness — Nassim Nicholas Taleb
  • Thinking, Fast and Slow — Daniel Kahneman

More from the shelf

N07 · Mind · 6 minInvert, Always InvertCharlie Munger once gave a graduation speech on how to be miserable. It is still the best advice he ever gave. N01 · Mind · 6 minAnd Then What?Three words that separate decisions that look smart today from decisions that still look smart in five years. O03 · Origins · 6 minWe Have Been Here BeforeAround 1450 a goldsmith in Mainz flooded the world with words. The people who thrived were not the ones you would guess.

See the whole library →