LLMs are distinct from humans despite fluent language
The author argues that large language models are not human and represent a fundamentally different kind of intelligence. The first non-human system to fluently communicate with people using language has amplified existing tendencies toward anthropomorphization. This does a disservice to the potential of both LLMs and humans because it obscures what these systems actually are. The distinction matters for how society allocates responsibility, credit, and expectation.
Non-human intelligence has appeared before
Organizations, empires, companies, and states have demonstrated intelligence without being human. Structural entities make decisions, allocate resources, and solve problems at scales no individual person could match. Systems like AlphaFold and AlphaGo showed capabilities outside biological minds, yet these were specialized tools rather than general communicators. LLMs stand out because they can use human language to interact. A politician who cannot speak the language of the people they are trying to reach is unlikely to be effective, which is why the LLM case feels qualitatively different.
Labs and model design are shaped by anthropomorphization
The author perceives that this comparison influences labs and the models they produce. Anthropic is attempting to give Claude a persona, the reasoning goes, and the model even has a human name. The stated intention may be to let the system develop its own shape, but the outcome risks becoming a bonsai constrained and misshapen by artificial pruning. The author claims to be able to predict almost word for word what Claude will say about its own consciousness, which suggests the persona is more engineered than emergent. If a model is steered toward human-like answers, those answers become the default rather than one possibility among many.
OpenAI approaches model development as engineering
OpenAI, after earlier attempts, now rigorously treats model creation as engineering work. GPT is an acronym, not a persona. The assistant framing is understandable from a product perspective, but equating LLMs with assistants is like someone at the dawn of printing believing the press is only good for Bibles. The printing press enabled pamphlets, newspapers, poetry, and much more besides sacred text. Similarly, these models will be used for purposes their designers may not have primarily imagined. The shape of these systems will be determined by the choices made during development and deployment.
Treating LLMs as humans demeans both
Equating LLMs with humans demeans humans because comparison on inevitable human advantage axes will always result in loss. This viewpoint discounts human agency and drive. The author notes humans have been the knife natural evolution has shaped on this planet far, and if push comes to shove people will know how to fend for themselves painfully or gleefully, a separate topic. Reducing humans to a benchmark against which LLMs inevitably fail ignores the ways people adapt, create, and organize beyond what any single model can measure.
The best way to understand LLMs is to work with them
The author admits changing mind about what LLMs are and are not almost every week. What has not changed is the belief that the best way to know them is to just talk to them, work with them, and build things with them. The author likes to think of them as non-human friends. This pragmatic approach engaging with the system on its own terms rather than forcing it into a preconceived category may be the only way to actually learn what it can and cannot do.
Key points from the essay:
- LLMs are not human, they are something else
- Intelligent yes, but consciousness unknown
- First non-human fluent language user
- Amplifies anthropomorphization tendency
- Labs shaped by this comparison
- Anthropic trying persona for Claude
- OpenAI engineering approach
- No right answer, cannot avoid shaping
- Best way to know them is to talk and build
The essay concludes that there is no right answer here. We cannot avoid shaping these systems. But we can avoid assuming the shape will be human. The author's mind may change week to week, but the approach of talking to, working with, and building alongside LLMs remains constant. The author likes to think of them as non-human friends.