LLMs can write code, review it, debug it, refactor it, and increasingly handle long chains of software tasks. The marginal cost of producing an intervention keeps collapsing. That is extraordinary. But it might also be quietly dangerous in a way nobody is measuring yet.
The crack repair machine problem
Picture a building with small cracks appearing in its walls. Historically, someone had to notice, investigate, and decide whether the crack was cosmetic or structural. That consumed human attention, which was scarce. Now imagine a machine that patches every crack instantly. Twenty cracks before lunch. The building looks fantastic.
The catch: some of those cracks were weak signals of load moving through the structure incorrectly. When local repair cost approaches zero, the feedback loop changes. Instead of investigating why a wall keeps cracking, the pattern becomes crack, patch, next. The building stays beautiful until the day the problem is not a crack. The load-bearing structure has moved.
This is the distribution-of-risk question that matters. The old world produced many small failures. They were visible, expensive, and annoying. But catastrophic failure was relatively rare because small failures continuously exposed structural weaknesses. Introduce machinery that makes local failure cheap, and the probability of small incidents drops. What happens to the probability of the tail event?
The scary scenario is not that AI writes bad code. That is boring. The scary scenario is that AI becomes extremely good at continuously repairing symptoms of structural weakness, reducing the human attention those symptoms previously attracted.
The diversity compression question
Consider a hundred humans. Each carries a different objective function. One wants money, another wants stability, another wants to make beautiful things, one wants status, one wants their children safe, one thinks the entire project is stupid. Humanity is a distributed system where everyone effectively gets a resource-allocation vote: spending one hour on this and zero hours on that. The friction between these incompatible goals looks inefficient, but it might carry structural information.
Now give those hundred humans LLM systems that perform a large portion of their intermediate cognitive work. What happens to the diversity of the resulting decisions? Do the hundred humans remain a hundred independent vectors, or does shared machinery introduce correlated blind spots? Does local productivity increase while systemic diversity decreases?
If the behavioral diversity through which different goals previously expressed itself gets compressed into a smaller effective space by layers of shared models, shared post-training, and shared interfaces, this stops being a productivity question. It becomes a stability question.
The missing logs problem
There is an information problem buried inside expertise that predates AI by centuries. An experienced practitioner observes patterns across decades: something looks wrong, feels wrong, sounds wrong. Feedback arrives. Weights update. Eventually a detection fires before the person can fully explain why. We call that experience, intuition, judgment.
The transferable documentation contains only fragments of the reasoning process. The next generation has to partly reconstruct the model by living through another enormous training run. This is simultaneously beautiful and horrifying engineering: an expert ships the model weights without the training data.
That information architecture becomes more urgent when the marginal cost of action drops. If throwing rocks is essentially free, sensing becomes more important, not less. The limiting resource stops being whether we can produce an intervention and becomes whether we know where to aim.
What gets expensive when action gets cheap
The AI era might be a sensing problem disguised as a generation problem. The field is obsessed with generation because generation is visibly improving. More code, more text, more agents, more actions, more automation. But if output becomes cheap enough, the person or system capable of saying stop, look at this one bit becomes disproportionately valuable.
Sensing, indexing, anomaly detection, goal preservation, knowing which bit matters, knowing whose objective function is being optimized, knowing when not to act. These shift from nice-to-have to critical. The question is not how much work the machine can do. The question is: when the machine makes action and repair almost free, what becomes expensive, and which weak signals stop receiving human attention because we no longer need humans to deal with the small failures?
The friction dividend
If billions of independently strange humans were part of the stabilizing feedback mechanism of civilization, what happens when increasingly large portions of their actions are mediated by a smaller family of machines? Maybe productivity goes up and everything works. Maybe humans simply move their attention one abstraction layer upward. Or maybe all the stupid friction, disagreement, duplicated work, weird intuition, and incompatible goals were carrying information that nobody realized was being used.
That is the hypothesis worth testing. Not whether AI is good or bad, not whether it counts as AGI. Give people the system, the objective, the environment, the constraints, the timestamps, the failures, the little cracks. Then change one thing and see what moves. If output becomes nearly free, the urgent question is whether anyone is getting better at sensing where to aim.