What Happens When a Company Tries to Scale Judgment? [On Judgment - Part 5]

Companies do not scale judgment by preserving it perfectly. They make parts of decision systems durable. The challenge is keeping rules, metrics and AI assessments valid and updatable once they gain operational force.

Amodiovalerio Verde

• 9 min read
What Happens When a Company Tries to Scale Judgment? [On Judgment - Part 5]

At the end of Part 4, I described this rather neatly: companies cannot put their best decision-maker everywhere, so some judgment gets turned into rules, standards and metrics. I still think that points at something real.

But I made the mechanism too clean.

In this series, I have used judgment narrowly: an assessment, not the preference, decision or authority that may follow from it. That distinction matters here.

There is a strange point in the life of a company where yesterday’s decisions become today’s infrastructure.

A team once debated what an “active user” should mean. Now it is a metric in a dashboard.

Someone once argued that a certain reliability level was acceptable. Now it is an SLO.

A security team once decided which conditions must be true before code can merge. Now they are branch protections.

A product team once chose which customers should see a capability. Now it is an entitlement rule.

A risk team once defined when a transaction deserves scrutiny. Now it is a threshold, model or workflow.

The people move on. The answers remain.

When I began researching this part of judgment, I had a neat theory for what was happening.

I thought organizations scaled judgment by encoding it.

A person forms a judgment. The company externalizes it into a process, metric, rule or system. The representation carries that judgment forward at scale.

It is a compelling story.

The more cases I looked at, the harder it became to defend as a general theory.

The hypothesis I expected to keep

The “encoding” idea appealed to me because it seemed to explain several familiar organizational problems: stale policies, distorted metrics, growing bureaucracy and inflexible software.

Yesterday’s assumptions become today’s constraints.

There is truth in that. But the theory assumes that a durable rule is basically a preserved human judgment.

Often it is not.

What becomes durable is not always judgment

Consider a product rule such as:

Accounts above a certain risk threshold require additional verification.

It might reflect a risk leader’s assessment, a compliance boundary, an incident response, statistical optimization, copied industry practice or twenty incremental changes made by different teams.

Often it is several of these at once.

More importantly, what becomes durable may not be a judgment at all.

It may be a standard. A preference. A prior decision. A constraint. An authority boundary.

And its provenance may be distributed across people, regulation, data, optimization and accumulated change.

Sometimes nobody could point to a single original assessment at all.

So companies do not literally scale judgment.

They make selected parts of a decision system durable.

That still creates some of the problems I was interested in.

Context can be lost.

Useful nuance can disappear.

But sometimes the context was never there in the first place.

And sometimes removing it is exactly what makes the rule useful.

Compression can be good

This was the correction I needed most.

I have a natural bias toward context.

Product leaders are trained to resist simplistic rules.

“Understand the customer.”
“Use judgment.”
“Do not let the framework decide.”

That instinct is useful.

It can become romantic.

There are many cases where repeated human discretion is worse than a well-designed rule:

  • Well-designed checklists can reduce omissions.
  • Security gates can prevent known classes of error.
  • Automated tests can apply the same standard every time.
  • Risk thresholds can process volumes no human team could inspect consistently.
  • A clear pricing boundary can be better than thousands of case-by-case negotiations.

Compression removes information.

But not all removed information was signal.

Sometimes context is noise, bias or an excuse for inconsistency.

The design problem is not “preserve all human nuance.”

It is more specific:

Which distinctions matter enough to survive, and which are better removed?

That is a harder question, but at least it describes the real work.

The more useful concept: operational force

Once I stopped trying to prove a general encoding theory, another distinction became more useful.

A metric, rule, model or policy can carry an assessment or standard.

What changes dramatically is what happens next.

So I started asking:

What can this assessment or standard actually cause?

Organizations do not just make assessments and standards reusable. They also make their consequences repeatable.

I call this operational force.

Operational force is not a property of the assessment itself. It describes how the organization connects an assessment or standard to action.

That connection can be direct or mediated. Sometimes the system itself applies the consequence: it routes the case, blocks the merge, changes the entitlement or executes the action.

Sometimes the force travels through people and institutions instead. A metric changes what leadership reviews. A target changes what teams optimize. A score affects how resources are allocated.

Both have operational force. But the mechanism is different.

In the first case, the assessment is coupled directly to action. In the second, it changes the conditions under which people make subsequent decisions.

An assessment can inform, rank, set a default, route, constrain, approve or block, or trigger execution.

A churn score shown on a dashboard informs.

The same score used to prioritize customer-success outreach routes work.

The same score used to automatically remove a discount executes an action.

Same assessment. Different operational force.

That does not mean the last system is necessarily more important or more dangerous. Operational force describes the coupling to action, not the magnitude of the consequence.

A model can be identical across all three systems. Governance should not be.

This is where ‘who made the judgment?’ is no longer enough. We also need to ask: ‘what can this thing do?’

When a standard becomes a gate

GitHub branch protection is a very concrete example of a standard becoming executable.

A team can require pull request reviews, passing status checks, successful deployments, signed commits, conversation resolution or other conditions before code can merge into a protected branch.

The standard might be something like:

Code entering main must meet our minimum quality and security conditions.

That sentence is broad.

The branch rule makes parts of it operational.

A required test is not advice. It can stop the merge.

A required review is not merely a recommendation. It can become a gate.

And GitHub also exposes something else that matters: bypass rights.

Who can override the rule?

Under which conditions?

Does the rule apply to administrators?

Which app is allowed to report a required status?

Those are governance questions expressed directly in product configuration.

Notice what we no longer need.

We do not need a senior engineer to personally reconsider the entire quality philosophy on every pull request.

That would be absurd.

The durable rule is valuable precisely because it turns a recurring standard into a reliable control.

The control is only as good as the standard it enforces, the checks used as proxies for that standard, and the ownership and bypass rules around it.

That is much closer to the real design problem.

When a trade-off becomes policy

Error budgets show the same mechanism at a different organizational level.

A service-level objective states an acceptable reliability target.

The error budget creates room for failure within that target.

Then a policy can connect the budget to release behavior.

Google’s published example policy is explicit: if a service exceeds its error budget over the preceding four-week window, all changes and releases are halted except for P0 or security fixes until the service is back within its SLO.

Look at what happened there.

A standard reflecting the acceptable trade-off between reliability and innovation became a repeatable operational constraint.

The company does not need an executive debate every Tuesday about whether this release is worth the reliability risk.

The rule carries the trade-off into the workflow.

This is not obviously a loss of useful discretion.

It may be a better decision system than repeated discretion.

But it still depends on assumptions.

Is the SLO right?

Does it represent the customer experience that matters?

Is the error budget calculated correctly?

Are some outages out of scope?

What happens when the architecture or user expectations change?

A durable standard is useful because it persists.

It becomes dangerous when persistence is confused with timeless validity.

Metrics can acquire force without looking like decisions

Software rules make operational force visible because they block things.

Metrics are more subtle.

Their operational force is often mediated rather than explicit.

A metric may appear to be descriptive.

Then incentives gather around it.

Roadmaps prioritize it.

Teams optimize it.

Leadership reviews revolve around it.

Models are trained against it.

Suddenly the metric is not merely measuring the system.

It is shaping the system.

In Part 3, I called this reflexivity: some systems partly create the evidence that later evaluates them.

Airbnb's search-ranking systems provide a concrete example. Ranking models use historical behavior to estimate which listings are more likely to be booked.

Those predictions influence which listings receive prominent exposure. Exposure then influences what guests see and book.

Taken together, that creates the conditions for a feedback effect. By changing visibility, the ranking system participates in producing some of the behavioral evidence future models may learn from.

Observed behavior is therefore not a neutral readout of preference. It is partly shaped by what the previous system chose to expose.

This is a general problem in deployed predictive systems, not something specific to Airbnb. Researchers describe related cases as performative prediction and algorithmic feedback loops: predictions and recommendations can change the behavior that later becomes training or evaluation data.

The important point is not that this loop is necessarily bad. It is that the system is no longer simply observing customer preference.

When booking probability becomes an important ranking objective, a choice about what to optimize acquires operational force.

It changes visibility, which changes behavior, which changes some of the evidence available to future systems.

That is more consequential than a model or dashboard choice.

It is system design.

The important questions are validity and staleness

Every durable standard has a scope, whether that scope is documented or not.

A fraud threshold calibrated on one customer mix may degrade when the mix changes.

A pricing boundary created for a startup may be absurd five years later.

A branch rule may become obsolete when the deployment architecture changes.

A metric definition may stop representing customer value after the product expands into a different use case.

The rule can continue functioning perfectly.

That is what makes staleness dangerous.

The system is not broken. It is faithfully executing an old assumption.

In Part 3, I called the surrounding property correctability: can the system preserve what it assumed, notice evidence that challenges it, and update without rewriting the past?

For durable organizational logic, staleness is the same problem at a different level.

A rule needs a recoverable scope and set of assumptions, and some way to notice when the product, customer, regulation, threat model, technology or business model has moved outside them.

The trigger might be a performance threshold, a new incident class, a material customer-segment change, a drift measure, a repeated override pattern or a scheduled review.

The goal is not constant reconsideration.

It is to design the update path before you need it.

That does not require a new governance artifact. Existing artifacts such as ADRs, model cards, metric dictionaries and configuration history can already preserve the rationale, scope, ownership and change history needed for later reassessment. If they do, use them.

A new concept is not automatically a new tool.

The goal is traceability, not another governance layer.

Not every durable premise began as one person’s judgment. Not every form of compression is loss.

What survived is not a theory of encoded judgment.

It is a design problem:

How do you keep durable organizational logic valid, contestable and updatable once it has operational force?

A five-question durability check

Pick one durable piece of product logic.

A threshold.
A ranking objective.
A permission.
A pricing-plan boundary.
A required workflow step.
A model output that changes what happens next.

Ask:

  1. What is this making durable: an assessment, standard, preference, decision or constraint?
  2. What can it cause?
  3. Where is it valid?
  4. Who can challenge, bypass or change it?
  5. What evidence triggers an update?

That is enough to uncover a surprising amount of hidden product logic.

Sometimes the answer will be reassuring.

The rule is still valid.

The scope is clear.

The update path works.

Leave it alone.

That negative case matters.

Persistence is not automatically technical debt.

A good rule that still works is an asset.

Then AI changes the question again

Everything so far could describe ordinary software.

A rule takes a defined input and applies a defined condition.

A metric computes a defined quantity.

A workflow routes a case.

Now introduce a system that can form a contextual assessment, generate a response and sometimes act on it at runtime.

It can assess risk.

It can classify intent.

It can inspect code.

It can recommend an action.

It can generate the action.

It can sometimes execute the action.

The old question “who decides?” becomes less useful on its own, because the best assessor may no longer be the person or system that should hold authority.

But there is now another separation to keep in view.

The same AI assessment could inform a human, route a case, set a default, block an action or execute one.

Those are different system designs even if the quality of the assessment is identical.

And the familiar answer, “keep a human in the loop,” turns out to hide as much as the word judgment did.

The next question is therefore uncomfortable:

If AI can form the better assessment in a defined class of cases, what should the human still do?

Sources