We usually argue that we should limit or intervene in the use of LLMs because we need human comprehension. But comprehension itself creates difficulty. The Chinese Room Argument was meant to fight strong functionalism and computationalism, yet it turns on a dubitable Cartesian premise: that genuine understanding consists in an isolated mental state with intrinsic intentionality. Wittgenstein's rule-following and Private Language Argument target this same premise, along with broader methodological solipsism. As the Stanford Encyclopedia of Philosophy synthesis notes, a language unintelligible to anyone but its creator is impossible—its originator could not establish meanings for its signs either. The issue is not comprehension. The issue is normativity.

When we say we should not hand everything over to LLMs, we make a normative claim. Neurobiological states or qualia, even if measurable, are normatively inert. The Chinese Room confuses a factual claim about understanding with a normative standing. Alone, neither a biological brain nor a self-attention block carries semantic normativity. In practice, we encounter normative objects—legal contracts, programs that produce right or wrong results. Normativity emerges from external social structures and communal practices. To say someone understands a concept is not to describe a private mental process but a social position: undertaking commitments and liabilities in the exchange of reasons.

This reframing explains why the Chinese Room feels unsettling and why human slop differs from AI slop. Humans are black boxes to one another, yet social structures and existential stakes bind them. Artificial tools lack such stakes, so they cannot function as full normative agents. Their societal integration relies on being auditable instruments, with human operators bearing responsibility. In modern deep learning, this auditability erodes. Massive pretraining on corpora that AI companies can acquire, combined with mysteriously interacting weights, creates epistemic opacity—the cognitive impossibility for human agents to inspect, decompose, or trace the warrant-generating elements of a computational process.

Overcoming epistemic opacity is not the Cartesian myth of seeking intrinsic semantics, which would relapse into private language solipsism. Rather it is pragmatic intelligibility: the public ability to grasp counterfactual dependencies, diagnose errors, and intervene. Compared to traditional tools, humanity faces normative disempowerment. We cannot hold the system as a peer, nor audit and revise it as a tool to the extent accountability demands.

Pharmacology offers a parallel dilemma. Medicine has long relied on empirically effective drugs, yet many lack full end-to-end understanding. The field achieves partial mechanical decomposability—for instance, AstraZeneca's approach using the 5R Framework—while strict clinical regulation and institutional liability shift skin in the game onto pharmaceutical companies and prescribing doctors. We cannot give weights genuine existential stakes, or the trouble would multiply. The hope is that mechanistic interpretation can bring LLMs to a place similar to pharmacology: a pragmatic middle ground where accountability is anchored in regulation and decomposability rather than inaccessible internal states.

The core tension reframes not as a comprehension gap but as a normative one. Human understanding is socially situated, accountable through commitments and liabilities. Artificial systems, stripped of existential stakes, integrate only as auditable instruments. Until interpretability methods anchor LLM behavior in inspectable, regulable structures, the disempowerment persists—we are neither peers to the system nor effective tools over it.