The Filter Was Running the Whole Time

[Views are my own.]

I got hundreds of those emails. The 419 scam. The dying widow, the Nigerian general, the prince who needed my bank account number to release his fortune. I deleted most without reading past the first line.

Most people did. That asymmetry was the product.

Cormac Herley, a researcher at Microsoft, studied why these scams advertised their implausibility so openly: the fabulous fortune, the West African setting, the premise almost everyone recognized immediately as fraudulent. His economic model suggests the implausibility was doing work. Follow-up required real human effort from the fraudster; a message that repelled careful readers and retained only the most credulous targets was more profitable than a polished message that attracted everyone. The implausibility was not a failure of execution. It was, in Herley's model, a selection mechanism.

The filter was running the whole time. Most people experienced it from the right side and never noticed it was there.

For a long time, I read AI slop as failure.

The filler prose, the numbered headers carrying no argument, the paragraphs that circle a point without landing, the 536th version of the same meme repackaged with different AI-generated filler, the short eye-catching video that has nothing to do with what you clicked on, the obvious absence of anything a real person observed or thought. My first reading was the obvious one: the producer tried to make something credible and produced something obviously synthetic.

That conclusion assumes concealment was the goal.

For some producers, the decisive threshold is not whether discerning readers consider the content good. It is whether the content clears a distribution system operating through imperfect behavioral and quality proxies: predicted click rate, dwell time, completion, share velocity. These systems attempt to measure quality and satisfaction, but quality must be encoded into signals, and signals can be decoupled from the thing they are supposed to represent. Content that converts on those signals is not slop from the platform's perspective. It is content that performed.

The human audience arrives afterwards, sorted by the algorithm. The segment that converts either does not notice the defects, does not care about them, or finds that noticing them is not enough to change its behavior. Improving quality would raise production cost without improving conversion among that audience. The visible defects may survive not because they were designed as a filter, but because removing them was never worth the cost.

At scale, the result can resemble deliberate filtering. Nobody had to design it. The system behaves as though they did.

Call this optimization filtering: the emergent property of a production and distribution system that rewards cheap output and measurable response. It becomes functional selection when the retained segment produces the relevant payoff and the excluded segment adds little the system values. Optimization systems can turn tolerated failure into functional selection.

The same logic applies to any system where quality must be encoded through proxies and production is cheap enough to decouple them: content platforms, product analytics, onboarding funnels, search.

This becomes uncomfortable when you apply it to products: an improving metric can conceal a deteriorating product if the system is becoming better at retaining users who tolerate the experience while losing those who value trust, control or depth. Defects can become locally optimal when fixing them raises cost without improving the measured outcome. And segmentation does not happen only through targeting: the experience itself selects the customers who remain. Every objective function is also a customer-selection mechanism.

Poor quality is not always an execution problem waiting to be fixed. Sometimes it is a locally rational outcome of the business model, production economics and measurement system. In those cases, the objective function, the incentives, or the distribution model has to change.

In Herley's model, the 419 scam used the mechanism deliberately. AI slop can produce a similar selection effect without requiring the same intent.

When discerning readers criticize AI slop publicly, they may still generate distribution-relevant activity. Hostile interaction can still register as a signal, even where platforms also model negative feedback and satisfaction. The producer does not need to read the critique. The platform already registered the interaction.

Appeals to quality enter a system that can register them only through proxies. Visibility, including hostile visibility, may still move the needle.

I write long articles. Each one takes weeks. At some point the question surfaces: whether that effort is justified when a thinner post with a clickable hook travels further and generates more engagement in three days than a careful argument does in three months.

The honest answer is: sometimes it does. I am not always winning this.

The ethical version, "does the end justify the means?", is the easier question to answer. I do not want readers asking whether what they are reading is the real version or the distribution version. That doubt, once planted, invalidates every piece before it. Reputation is not separate from content quality. It is the same asset, accumulated slowly and destroyed fast.

The strategic version is harder. Careful work and good distribution are different disciplines. The second does not follow from the first, and the optimization filter may push discerning readers away from the high-volume feed, but it does not automatically deliver them to your door. Reaching them still requires the feed to work at least once.

That is the part I have not solved.

When recognition makes you leave, you were sorted out of the audience the system had learned to retain. If the objective function does not represent the value you would have brought, your departure will not register as failure. In that case, the filter found someone it did not know how to value.

Do not judge a system by how obvious its defects are. Judge it by whether those defects are costly to the outcome the system actually optimizes.

A direct subscription channel works differently. The subscriber may have found you through an algorithm, but future delivery no longer depends on each article winning the feed again. They chose to receive it. That is a different kind of attention, and it compounds in a way that feed reach rarely does.

The alternative to playing the volume game is not to win the feed by becoming more like it. It is to build a form of distribution where judgment is part of the objective function.

Sorted out is not the same as left behind. The harder question, the one I have not answered, is whether building on the right side of the filter is a strategy or a consolation. Most people creating careful content are sitting with that same calculation. Some have probably resolved it differently.

Reference: Cormac Herley, “Why Do Nigerian Scammers Say They Are from Nigeria?”, Microsoft Research, 2012.