Statistical Illusions Entry #0449 Classified Declassified

The reason a benchmark becomes a distortion

A published comparison reorganises the thing it measures. Effort moves to the scored dimensions, and the unscored ones become where the cost is paid.

No visual record attached The written record below is complete.
Plate 499 — The ranking nobody admits to using

Intuition test — answer before you read on

Why does a published benchmark change the activity it measures?

The ranking is published annually and everyone insists they ignore it. Four years later the admissions criteria, the spending priorities and the internal reporting have all been rebuilt around its components, and nobody remembers deciding to do that.

What everyone sees

A benchmark is treated as an observation — a description of performance that leaves performance unchanged. Publication makes it an intervention. Once a score has consequences, the score becomes an objective, and the parts of the activity it does not cover lose the resources that flow to the parts it does.

What is actually happening

Espeland and Sauder studied law school rankings and documented reactivity in detail: institutions reorganised admissions, spending and reporting around the ranked components, administrators described feeling compelled by a measure they considered invalid, and the ranking’s categories became the language in which quality was discussed. Muller’s survey of metric-driven management gathers the general pattern across sectors — short-termism, goal displacement and effort diverted from unmeasured work — while noting that measurement is not the problem and the substitution of measurement for judgement is. The 2026 case that makes it vivid is model evaluation: leaderboard scores now shape research agendas, and contamination of test sets means the same number is both the target and the compromised instrument.

Why it stays hidden

The reorganisation hides because it happens through ordinary decisions. Nobody adopts the benchmark as a goal; individual choices are made about admissions, hiring and reporting, each defensible on its own, and each incidentally aligned with the scored dimensions. The unscored work degrades quietly, and there is no series recording it.

Publishing a comparison makes it an intervention. Effort migrates to the scored dimensions, and the unscored ones have no series to show the loss.

Publishing a comparison makes it an intervention. Effort migrates to the scored dimensions, and the unscored ones have no series to show the loss.

The hidden part — entry #0449

Collect this card

Publishing a comparison makes it an intervention. Effort migrates to the scored dimensions, and the unscored ones have no series to show the loss.

0 / 10,000 collected

Sources & further reading 2
  1. Espeland & Sauder — rankings and reactivity: how public measures recreate social worlds
  2. Muller — the tyranny of metrics

Circulate this file

Annotations are reserved for archive members.

Sign in to annotate