Statistical Illusions Entry #0431 Classified Declassified

The reason a survey of survivors misses the finding

Study only the cases that remain and the causes of disappearance become invisible. The missing rows are usually the ones carrying the answer.

No visual record attached The written record below is complete.
Plate 108 — a dataset, with the excluded rows sketched in outline.

Intuition test — answer before you read on

A study of long-lived firms finds six shared practices and calls them causes of longevity. What is the flaw?

A study of long-lived firms identifies six shared practices and publishes them as a formula. Ten years later a substantial share of those firms have shrunk or gone. The study never examined the firms that followed the same six practices and failed, because those firms were not in the population it drew from.

What everyone sees

A sample of successes looks like the right place to study success. The reasoning is intuitive: examine the outcome you want, find what its holders share, recommend it. The step that is skipped is checking whether the same features are equally common among the cases that did not survive.

What is actually happening

Survivorship bias is a selection effect: the sampling frame is defined by the outcome, so any feature that failures also possess will appear causal. Wald’s wartime analysis of returning aircraft made the structure explicit — damage patterns on survivors indicate where hits are survivable, not where they occur. Denrell showed formally that studying only survivors systematically overstates the value of risky strategies.

Why it stays hidden

The absent cases hide because absence is not visible in a dataset. Failed firms stop filing, stop being listed and stop being interviewed, so the surviving records look complete. Narrative pressure completes the concealment: a study of survivors produces actionable advice, while a study including failures often produces the finding that the feature was not decisive, which is harder to publish and harder to sell.

The sampling frame is the finding. If the population was selected by outcome, shared features prove nothing about cause.

The sampling frame is the finding. If the population was selected by outcome, shared features prove nothing about cause.

The hidden part — entry #0431

Collect this card

The sampling frame is the finding. If the population was selected by outcome, shared features prove nothing about cause.

0 / 10,000 collected

Sources & further reading 3
  1. Wald, "A Method of Estimating Plane Vulnerability Based on Damage of Survivors", Statistical Research Group, 1943
  2. Denrell, "Vicarious Learning, Undersampling of Failure, and the Myths of Management", Organization Science, 2003
  3. Brown et al., "Survivorship Bias in Performance Studies", Review of Financial Studies, 1992

Circulate this file

Annotations are reserved for archive members.

Sign in to annotate