Why a p-value is not a probability that you are right
The p-value measures how surprising the data would be if the null hypothesis were true. It says nothing about the probability that the hypothesis itself is true.
Filed by The Archivist 2 min read
Intuition test — answer before you read on
A study reports p = 0.03. A reader infers a 97 per cent chance the finding is real. What went wrong?
Correct answer: B
Option A addresses a real problem but not this one. The error is the conditional inversion: the p-value measures how surprising the data is under the null, not the probability that the null is false. Converting it to the latter requires a prior that the p-value does not supply.
A study reports p equals 0.03 and concludes the result is statistically significant. A reader interprets this as a 97 per cent chance the finding is real. The interpretation feels intuitive, direct and wrong. The p-value is a statement about the data given a hypothesis, not a statement about the hypothesis given the data.
What everyone sees
The number 0.03 looks like a probability and is presented alongside a yes-or-no decision about significance. The natural reading is that 0.03 is the probability the result happened by chance, and therefore 0.97 is the probability it is real. This reading is the most common misinterpretation in published science and in journalism about science.
What is actually happening
The p-value is defined as the probability of obtaining data at least as extreme as the observed data, assuming the null hypothesis is true. It is P(data | null), not P(null | data). Inverting the conditional requires Bayes’ theorem and a prior probability of the hypothesis, which the p-value does not contain. Goodman showed that the same p-value can correspond to very different posterior probabilities depending on the prior. A p of 0.03 in a well-powered replication of a plausible hypothesis means something very different from a p of 0.03 in an exploratory study of an implausible hypothesis, but the number looks the same.
Why it stays hidden
The misreading hides because the conditional inversion is invisible in everyday language. “The probability of this data if the hypothesis is false” sounds almost identical to “the probability that the hypothesis is false given this data,” and the difference between the two requires formal probability training to distinguish. Most readers, including many researchers, process the number as the latter because it answers the question they actually want answered.
A p-value tells you how surprising the data is. It does not tell you how true the hypothesis is.
A p-value tells you how surprising the data is. It does not tell you how true the hypothesis is.
Collect this card
A p-value tells you how surprising the data is. It does not tell you how true the hypothesis is.
0 / 10,000 collected
Sources & further reading 2
- Goodman — a dirty dozen: twelve p-value misconceptions
- Greenland et al. — statistical tests, p-values, confidence intervals, and power
Cross-references
Related files
Filed near this one in the index.
-
No visual on fileScarcity & Queues Entry #0461
Why a paywall creates a different reader
The paywall does not merely filter who reads. It changes how the remaining readers read, because the cost they paid becomes part of the experience.
NoviceThe hidden part #0461The paywall does not select better readers. It makes the same readers better at reading.
Scarcity Open file -
No visual on fileScarcity & Queues Entry #0457
Why a members-only door raises the perceived quality inside
A restriction implies that something is being protected. The inference is made before anything inside has been seen, and it survives finding nothing there.
NoviceThe hidden part #0457Restriction and value are separable. The door can be installed first, and the perceived quality assembles itself behind it.
Scarcity Open file -
No visual on fileScarcity & Queues Entry #0451
Why a limited edition raises value without raising quality
The product is identical. The only thing that changed is the number that will be made, and that number changed the price people are willing to pay.
NoviceThe hidden part #0451Supply constraint is a design decision. The limited edition does not discover rarity; it manufactures it and lets the market treat the result as worth.
Scarcity Open file -
No visual on fileStatistical Illusions Entry #0432
Why a correlation of one variable with time explains nothing
Two quantities that both rise over the years will correlate strongly with each other. The shared trend does the work and neither variable has touched the other.
NoviceThe hidden part #0432Two trends will always agree. The coefficient reports the shared direction of time, not a relationship between the variables.
Statistics Open file
Annotations are reserved for archive members.
Sign in to annotate