The reason a benchmark becomes a distortion
A published comparison reorganises the thing it measures. Effort moves to the scored dimensions, and the unscored ones become where the cost is paid.
Filed by The Archivist 2 min read
Intuition test — answer before you read on
Why does a published benchmark change the activity it measures?
Correct answer: B
Option A occurs but the effect is present without any deception. Option C is a design critique rather than the mechanism. Ranked institutions reorganised admissions, spending and reporting around ranked components while regarding the measure as invalid, and metric-driven management produces goal displacement and diverted effort across sectors.
The ranking is published annually and everyone insists they ignore it. Four years later the admissions criteria, the spending priorities and the internal reporting have all been rebuilt around its components, and nobody remembers deciding to do that.
What everyone sees
A benchmark is treated as an observation — a description of performance that leaves performance unchanged. Publication makes it an intervention. Once a score has consequences, the score becomes an objective, and the parts of the activity it does not cover lose the resources that flow to the parts it does.
What is actually happening
Espeland and Sauder studied law school rankings and documented reactivity in detail: institutions reorganised admissions, spending and reporting around the ranked components, administrators described feeling compelled by a measure they considered invalid, and the ranking’s categories became the language in which quality was discussed. Muller’s survey of metric-driven management gathers the general pattern across sectors — short-termism, goal displacement and effort diverted from unmeasured work — while noting that measurement is not the problem and the substitution of measurement for judgement is. The 2026 case that makes it vivid is model evaluation: leaderboard scores now shape research agendas, and contamination of test sets means the same number is both the target and the compromised instrument.
Why it stays hidden
The reorganisation hides because it happens through ordinary decisions. Nobody adopts the benchmark as a goal; individual choices are made about admissions, hiring and reporting, each defensible on its own, and each incidentally aligned with the scored dimensions. The unscored work degrades quietly, and there is no series recording it.
Publishing a comparison makes it an intervention. Effort migrates to the scored dimensions, and the unscored ones have no series to show the loss.
Publishing a comparison makes it an intervention. Effort migrates to the scored dimensions, and the unscored ones have no series to show the loss.
Collect this card
Publishing a comparison makes it an intervention. Effort migrates to the scored dimensions, and the unscored ones have no series to show the loss.
0 / 10,000 collected
Sources & further reading 2
- Espeland & Sauder — rankings and reactivity: how public measures recreate social worlds
- Muller — the tyranny of metrics
Cross-references
Related files
Filed near this one in the index.
-
No visual on fileStatistical Illusions Entry #0433
The reason a small sample produces the biggest effects
Rank any set of groups by outcome and the smallest ones fill both extremes. Low counts produce wide swings, and the swing is mistaken for the finding.
AdeptThe hidden part #0433Small samples swing hardest in both directions. Publishing only one tail turns the swing into a recommendation.
Statistics Open file -
No visual on fileStatistical Illusions Entry #0435
The reason a graph without zero can double an effect
Truncating the y-axis does not change the data. It changes the slope the eye sees, and the eye reads slope as magnitude before the mind reads the numbers.
MasterThe hidden part #0435A graph with correct data and a truncated axis is not lying about the numbers. It is lying about the shape.
Statistics Open file -
No visual on fileScarcity & Queues Entry #0460
The reason rationing changes what people want
A product that was adequate becomes desirable when restricted. The restriction changed the inference the buyer draws, not the product itself.
MasterThe hidden part #0460A quota does not just limit access. It changes the inference about what is being accessed. The product behind the restriction is perceived as the product that deserved the restriction.
Scarcity Open file -
No visual on filePrice Psychology Entry #0360
The reason a currency without symbols raises spend
Remove the dollar sign and the number stops feeling like money. It becomes a score, and scores are spent differently from budgets.
MasterThe hidden part #0360A currency symbol is not information. It is a pain cue. Remove it and the number stays; the flinch goes.
Pricing Open file
Annotations are reserved for archive members.
Sign in to annotate