The reason a survey of survivors misses the finding
Study only the cases that remain and the causes of disappearance become invisible. The missing rows are usually the ones carrying the answer.
Filed by The Archivist 2 min read
Intuition test — answer before you read on
A study of long-lived firms finds six shared practices and calls them causes of longevity. What is the flaw?
Correct answer: B
Sample size is not the issue. Selecting the population by the outcome means any trait shared by failures too will appear causal. Wald’s aircraft analysis made this structural: survivor damage shows where hits are survivable, not where they land.
A study of long-lived firms identifies six shared practices and publishes them as a formula. Ten years later a substantial share of those firms have shrunk or gone. The study never examined the firms that followed the same six practices and failed, because those firms were not in the population it drew from.
What everyone sees
A sample of successes looks like the right place to study success. The reasoning is intuitive: examine the outcome you want, find what its holders share, recommend it. The step that is skipped is checking whether the same features are equally common among the cases that did not survive.
What is actually happening
Survivorship bias is a selection effect: the sampling frame is defined by the outcome, so any feature that failures also possess will appear causal. Wald’s wartime analysis of returning aircraft made the structure explicit — damage patterns on survivors indicate where hits are survivable, not where they occur. Denrell showed formally that studying only survivors systematically overstates the value of risky strategies.
Why it stays hidden
The absent cases hide because absence is not visible in a dataset. Failed firms stop filing, stop being listed and stop being interviewed, so the surviving records look complete. Narrative pressure completes the concealment: a study of survivors produces actionable advice, while a study including failures often produces the finding that the feature was not decisive, which is harder to publish and harder to sell.
The sampling frame is the finding. If the population was selected by outcome, shared features prove nothing about cause.
The sampling frame is the finding. If the population was selected by outcome, shared features prove nothing about cause.
Collect this card
The sampling frame is the finding. If the population was selected by outcome, shared features prove nothing about cause.
0 / 10,000 collected
Sources & further reading 3
- Wald, "A Method of Estimating Plane Vulnerability Based on Damage of Survivors", Statistical Research Group, 1943
- Denrell, "Vicarious Learning, Undersampling of Failure, and the Myths of Management", Organization Science, 2003
- Brown et al., "Survivorship Bias in Performance Studies", Review of Financial Studies, 1992
Cross-references
Related files
Filed near this one in the index.
-
No visual on fileStatistical Illusions Entry #0437
The reason a study that replicates is worth two that do not
A single study is a claim. A replication is a test of that claim. Two unreplicated studies are two untested claims, not twice the evidence.
AdeptThe hidden part #0437Novelty fills journals. Replication fills knowledge. The incentive structure rewards the wrong one.
Statistics Open file -
No visual on fileStatistical Illusions Entry #0434
Why a percentage change needs its starting point
A fifty per cent increase from two is one. A fifty per cent increase from two million is one million. The percentage is identical; the meaning is not.
AdeptThe hidden part #0434A percentage without a base is not a statistic. It is a frame looking for a denominator the reader will supply.
Statistics Open file -
No visual on fileScarcity & Queues Entry #0462
The reason an invitation raises acceptance more than an offer
An offer says the item is available. An invitation says the recipient was selected. The difference is not in the item but in the identity it assigns.
AdeptThe hidden part #0462An offer presents a product. An invitation presents an identity. People accept identities faster than products.
Scarcity Open file -
No visual on fileScarcity & Queues Entry #0458
The reason an expiring discount beats a larger permanent one
A permanent reduction can be acted on at any time, which means it can be postponed indefinitely. A deadline converts an intention into a dated task.
AdeptThe hidden part #0458An open offer can be postponed forever. The deadline is not pressure; it is the thing that gives the intention a date.
Scarcity Open file
Annotations are reserved for archive members.
Sign in to annotate