A widely shared blog post this week laid out a detailed case for why large language models will remain useful tools rather than autonomous workers, even after high-profile breakthroughs in mathematics and security research. The argument, published on dank.systems, goes beyond the usual "LLMs can't do X" take and instead examines the structural economics of when and where AI agents actually make sense.
The Navier-Stokes problem is not the norm
The recent wave of LLM achievements in formal mathematics, like solving problems related to the Navier-Stokes equations, and finding real-world security vulnerabilities has fueled expectations that agents are close to replacing knowledge workers. The post argues these results are misleading precisely because they represent the best possible conditions for agentic work. A mathematical theorem statement is already a rigorous specification. It has been audited by the mathematical community for decades. The Lean theorem prover used to verify proofs was designed specifically to resist the kind of reward hacking that plagues LLM training. Even with all of that infrastructure in place, soundness bugs in proof checkers have allowed LLMs to launder bogus proofs through the kernel before.
The vast majority of knowledge work does not look like theorem proving. Most tasks lack formal specifications entirely, and creating them requires a rare combination of domain expertise and specification engineering skill that even many experienced software developers do not possess.
The specification bottleneck is real and expensive
The post draws on hardware engineering as a concrete example. A typical CPU design project employs roughly three times as many specification and validation engineers as design engineers, with some teams running ratios as high as five to one. Writing a specification that an automated system can implement correctly is itself a specialized skill, separate from understanding the problem domain. For many tasks, specifications must evolve alongside implementation, making a one-time spec-and-forget approach impossible.
This creates a fundamental tension. If you need rigorous specifications to prevent reward hacking, and those specifications cost more to produce than just doing the work manually, the economic case for full automation collapses in most domains. The only exceptions are fields where failure at deployment is catastrophic, such as chip design or drug discovery, where expensive specification and validation is already standard practice.
Human review does not scale either
The obvious fallback when you cannot write a formal specification is human review. But human review hits its own ceiling. It does not scale to the volume of output a language model can produce. Worse, expert human reviewers are themselves vulnerable to sophisticated attacks. The post cites the XZ Utils backdoor, where a malicious contributor spent years building trust before inserting a backdoor into a critical Linux utility, and the University of Minnesota hypocrite commits incident, where researchers submitted intentionally flawed patches to the Linux kernel to prove a point about code review processes.
If human review remains a required step in the production loop, the speed of the entire system is constrained by how fast humans can read and evaluate output. That is the opposite of the "country full of geniuses in a datacenter" narrative that frontier lab CEOs promote.
Three categories where autonomous AI actually works
The post identifies exactly three types of organizations that can benefit from fully autonomous LLM use. First, those where failure is cheap: firms doing rapid prototyping or work where an intern-level mistake costs little to fix. Second, those with a small set of narrowly defined tasks that already have clear guardrails, like repetitive physical labor in controlled environments or call center operations. Third, those that already accept the costs of rigorous specification and validation, such as chip designers and pharmaceutical researchers.
The first two categories are price-sensitive and do not need frontier model capabilities. Open models running on cheap hardware, potentially even on-premises, serve them better. The third category might still use frontier models, but the post argues that even their work could be handled by cheaper alternatives like DeepSeek V4.1 Flash, and that swarm-based approaches using many small models may outperform single large models for combinatorial search tasks.
The data center full of brainlets problem
The post closes with a pointed observation about the two competing visions for AI's future. In the artificial superintelligence scenario, a self-improving system drives its own progress, limited only by available compute. In the "data center full of geniuses" scenario that frontier labs sell, the system is actually composed of many limited models that need human orchestration at every step. The bottleneck in the second scenario is not compute but human attention and time. The author's prediction: the blast radius of this structural limitation will extend well beyond the frontier labs themselves, affecting the broader industry's expectations about what AI can deliver on a realistic timeline.