The reason peer review rewards caution
A reviewer who wrongly rejects is never identified. A reviewer who wrongly accepts is named in the record, and the two errors are not priced alike.
Filed by The Archivist 2 min read
Intuition test — answer before you read on
Why does peer review tend to favour cautious over exploratory work?
Correct answer: B
Option A appears in specific disputes but not as a general pattern. Option C describes a correlation without a mechanism. Resubmitted proposals reviewed by fresh panels produced agreement near chance levels for many decisions, and the costs of the two error types fall asymmetrically — one anonymous, one permanently attributable.
Two manuscripts arrive with the same referee. One is careful, incremental and certain to be cited eleven times. The other is odd, possibly wrong and possibly important. The referee has an hour, no payment and an asymmetric exposure, and the odd one goes back for major revisions.
What everyone sees
Conservatism in review is read as intellectual temperament — a matter of taste in what counts as sound work. Temperament varies between individuals; the tilt does not. It appears across fields, panels and career stages, which points at the structure of the task rather than at the disposition of the people performing it.
What is actually happening
Cole, Cole and Simon resubmitted funded and unfunded proposals to fresh reviewer panels and found agreement close to what chance would produce for a large share of decisions, indicating that outcomes depend heavily on which reviewers were drawn rather than on a stable property of the proposal. Nosek, Spies and Motyl set out the incentive structure around that noise: the costs of the two possible errors fall in different places. A wrongly rejected paper leaves no trace attributable to anyone, since it is either published elsewhere or never seen. A wrongly accepted paper that fails is a durable, searchable object attached to a journal and, informally, to the people who let it through. Facing noisy signals and asymmetric costs, the risk-minimising response is to demand more evidence.
Why it stays hidden
The asymmetry hides because rejection is framed as rigour, and rigour is a virtue rather than a position. Nothing in a referee report says that the risk of being associated with an error exceeded the risk of missing something. The reasons given are always about the work, and they are usually accurate reasons that would also have applied to accepted papers.
The two errors of review are not priced alike. A missed discovery is anonymous and a published mistake is attributable, so caution is the cheaper mistake.
The two errors of review are not priced alike. A missed discovery is anonymous and a published mistake is attributable, so caution is the cheaper mistake.
Collect this card
The two errors of review are not priced alike. A missed discovery is anonymous and a published mistake is attributable, so caution is the cheaper mistake.
0 / 10,000 collected
Sources & further reading 2
- Cole, Cole & Simon — chance and consensus in peer review
- Nosek, Spies & Motyl — scientific utopia II: restructuring incentives and practices
Cross-references
Related files
Filed near this one in the index.
-
No visual on fileStatistical Illusions Entry #0436
Why a p-value is not a probability that you are right
The p-value measures how surprising the data would be if the null hypothesis were true. It says nothing about the probability that the hypothesis itself is true.
NoviceThe hidden part #0436A p-value tells you how surprising the data is. It does not tell you how true the hypothesis is.
Statistics Open file -
No visual on fileScarcity & Queues Entry #0461
Why a paywall creates a different reader
The paywall does not merely filter who reads. It changes how the remaining readers read, because the cost they paid becomes part of the experience.
NoviceThe hidden part #0461The paywall does not select better readers. It makes the same readers better at reading.
Scarcity Open file -
No visual on fileScarcity & Queues Entry #0457
Why a members-only door raises the perceived quality inside
A restriction implies that something is being protected. The inference is made before anything inside has been seen, and it survives finding nothing there.
NoviceThe hidden part #0457Restriction and value are separable. The door can be installed first, and the perceived quality assembles itself behind it.
Scarcity Open file -
No visual on fileScarcity & Queues Entry #0451
Why a limited edition raises value without raising quality
The product is identical. The only thing that changed is the number that will be made, and that number changed the price people are willing to pay.
NoviceThe hidden part #0451Supply constraint is a design decision. The limited edition does not discover rarity; it manufactures it and lets the market treat the result as worth.
Scarcity Open file
Annotations are reserved for archive members.
Sign in to annotate