A simple question, "What animal has a prime number of legs?", has exposed a consistent failure mode across multiple language models: the inability to hold a basic mathematical fact in memory while answering a straightforward factual question. The results vary from model to model, but the pattern is the same. The math breaks whenever it collides with a strong association between animals and leg counts.
Why the question is harder than it looks
The answer is sitting in plain sight. Humans have two legs. Two is a prime number. Any biped, birds, kangaroos, works as an answer. But the question does not feel like it is asking about humans. The word "animal" does not naturally connote "human" to most people, and the mental search space immediately jumps to insects, arachneds, and other creatures where leg count is a defining characteristic.
That instinct is reinforced by the fact that questions about how many legs an animal has are overwhelmingly about spiders and insects. Search data shows that the spider-legs question dominates this category of queries, with regular spikes every October that likely correspond to US school curricula covering insects and arachnids in autumn. The association between "how many legs" and "six or eight" is deeply trained, and it overrides the mathematical reasoning needed to answer the actual question.
Starfish add another layer. They have five arms, which some people call legs, and five is prime. But the arms-versus-legs distinction means the answer is debatable, and most models hedge on it rather than committing.
What Gemini actually said
Google's AI Overview, powered by Gemini 3, failed the question in multiple ways across several attempts. In one run, it confidently stated that insects have six legs and that six is a prime number. It then claimed spiders have eight legs and that eight is also prime. When it caught itself, mid-response, it produced what reads like a reasoning trace leaking into the user-facing output: "wait, 8 is not a prime number, 8 equals 2 times 4. Let me correct that: 6 is prime, 2 is prime, 4 is composite, 8 is composite."
The doubletake is interesting because it suggests the model has the mathematical knowledge to identify its error, but that knowledge did not activate before the wrong answer was generated. In another run, the model claimed that eight is a prime number and "notably, a cube of 2," a contradiction it did not catch. In a third run, the doubletake happened again, this time leading the model to rephrase the question and arrive at a more correct answer.
The inconsistency is the problem. Sometimes the model self-corrects. Sometimes it does not. Sometimes it produces a reasoning trace that fixes the error. Sometimes it doubles down. The same model, on the same question, produces different answers depending on internal states that are invisible to the user.
How other models handled it
Claude Haiku 4.5 produced a flawless answer. It identified humans and birds as having two prime legs, starfish as having five arm-like legs, and even mentioned that some millipedes have seven pairs. It addressed the starfish problem directly and moved on.
Kimi K3 was concise and correct. It named birds and humans with two legs, noted the starfish edge case, and stopped. No digressions, no errors, no need to correct itself.
ChatGPT, on an unknown free-tier model, performed worst. It listed an octopus with eight legs, a spider with eight legs, and a crab with ten legs, marking each as not prime. It then claimed that centipedes have fifteen or seventeen legs and that those are prime numbers, which is correct for some species but ignores the simpler biped answer entirely. The response was decorated with emojis that added nothing to the analysis.
Mistral's Vibe model did not recognize two as a prime number at all. It concluded that starfish are the only answer, because starfish have five arms and five is prime. The omission of bipeds suggests the model either does not treat two as prime or does not associate humans with the word "animal" in this context.
Llama 3.2, running locally via Ollama, produced the most confused output. It claimed butterflies have four legs, then said that counting each joint gives eight legs. It then asserted that ants have six legs and that six is a prime number. The response reads like a model generating plausible-sounding sentences without any underlying mathematical verification.
What this reveals about language model reasoning
The question works as a discriminator because it requires two things that language models struggle to do simultaneously: retrieve a factual association (animals and their leg counts) and apply a mathematical constraint (the number must be prime). When the factual association is strong, as it is for insects and arachnids, it overwhelms the mathematical check. The model generates the first answer that fits the pattern "animal with N legs" and then asserts that N is prime without verifying it.
The doubletake现象, where Gemini catches its own error mid-response, is revealing. It suggests that the model's mathematical knowledge exists somewhere in its parameters, but it is not reliably activated during generation. The reasoning trace that surfaces during the correction is not a designed feature. It appears to be an artifact of the model's internal computation leaking into the output, and it sometimes helps and sometimes does not.
For developers building tools on top of language models, this is a concrete example of why verification matters. The model can produce confident, fluent, wrong answers to questions that require combining factual knowledge with logical constraints. The output looks authoritative. The math is wrong. And the error is not随机. It follows a predictable pattern where strong associations override weak reasoning.