Reading Path 05 · When Metrics Become Mechanisms
Metrics are often treated as neutral representations of an underlying reality.
The writing increasingly challenges that assumption.
A metric can be incomplete without being dangerous. The more consequential shift happens when the representation starts affecting the system itself: teams optimize for what is visible, models learn from the data their earlier choices helped produce, platforms assign scores that influence treatment, and classifiers begin standing in for concepts they cannot actually observe.
This path follows that move from weak proxies to active mechanisms.
It begins with the difference between motion and progress. It then develops through measurement design, self-reinforcing feedback loops and optimization effects, before moving into hidden scores, unreliable classifiers and a final challenge to one of the strongest proxies of all: using outcomes as proof that an earlier judgment was good.
Shipping fast isn't progress. Clean backlogs and on-time releases can mask weak impact. When strategy is a slide deck, and success means 'shipped', teams sprint in circles. Progress is learning, behavior change, and clarity. Ask: what are we trying to learn, change, and prove? Then adjust. Not ship.
The Illusion of Progress →AI now shapes product strategy, not just predicts it. Left unquestioned, it becomes a prophecy engine: reinforcing biases, narrowing options, and derailing learning. Treat AI as input, test counterfactuals, review second-order effects, and keep humans in the reasoning loop. Question it. Validate it.
When AI Turns Your Product System Into a Self-Fulfilling Prophecy →In the Data Delusion, we focus on metrics while losing sight of what truly matters. We celebrate tiny, irrelevant wins, creating an illusion of progress. Data without human judgment is just noise – it's time to use it to sharpen our questions, not just track our speed.
Beyond the Dashboard | Principle 1: Avoid the Data Delusion →Think like a doctor, not a data collector. Your dashboards should be a cockpit, not a buffet. Every metric has a cost in attention, fueling debate and cognitive load. Track only what informs decisions. If a number doesn't drive action, it's just noise.
Beyond the Dashboard | Principle 3: Choose What to Measure →AI slop may not be failing. It may be doing exactly what the system rewards: filtering for audiences who tolerate it. What looks like poor execution can become a rational outcome of the objective function, incentives and distribution model.
The Filter Was Running the Whole Time →I found a file in my LinkedIn data export I could not explain. Ten categories. A credibility score. A notability tier. Twenty-one dated snapshots, all identical. I published throughout that period. Nothing moved.
The Credibility Score →AI writing detectors can misclassify polished human writing. A practical look at style, authorship and the limits of detection.
The Detector Doesn't Know Cicero →If judgment is an assessment, the harder question is how to tell whether it was a good one. Outcomes help, but they can also mislead.
Was It Good Judgment, or Did It Just Work? [On Judgment - Part 3] →Further reading