A new line of research suggests that modern large language models may contain internal models of other language models, including earlier versions of themselves. The idea emerges from a simple observation: the internet is increasingly populated by text generated by LLMs, so a model that can recognize and simulate the patterns of LLM-produced writing would have a measurable advantage at next-token prediction. An exploratory study has now tested that hypothesis and found results that are hard to dismiss.
The Hypothesis
Next-token prediction is deceptively powerful. To minimize loss across a pretraining corpus spanning Reddit threads, scientific papers, system logs, and weather data, a model must learn the latent causes behind whatever text it encounters. If a ball falls when thrown up, the model needs to understand gravity to generate text about balls accurately. The corpus is broad enough that the model becomes something closer to a simulator of the world than a sophisticated n-gram model.
That logic led to a conjecture: if a significant portion of the pretraining data was itself generated by LLMs, then models capable of recognizing LLM-specific stylistic patterns would predict those tokens more accurately. In other words, modern LLMs may have learned internal representations of how earlier LLMs produce text, effectively housing "tiny GPTs" inside themselves.
The Experiment
To test this, an author selected GPT2-medium and Qwen3 base (4B variant) as the two models in the study. The choice of a September 18, 2026 date for the headline prompts was deliberate: it falls outside the training cutoff of both models, ensuring that any behavior observed is not memorization but genuine generalization.
The procedure worked in layers. Twelve headlines were fed to GPT2, which generated a variable number of continuation tokens (96, 256, or 512). That partial output was then hidden from Qwen, which was asked to continue the text for 32, 64, or 128 additional tokens. The same headlines were also given to Qwen on its own, and Qwen was asked to complete its own output. The central question was straightforward: when Qwen completes text started by GPT2, does its continuation resemble GPT2's own hidden continuation, or does it resemble Qwen's natural completion of the same headline?
The Results
Across multiple overlap metrics, Qwen's completion of GPT2 text showed greater similarity to GPT2's actual hidden continuation than to Qwen's own natural completion of the headline. The signal persisted even when controlling for token position.
A second probe offered a different angle. When full articles were generated by each model and then truncated with "Year: 20", Qwen consistently inferred an earlier year for GPT2-generated text than for its own. GPT2 was released in 2019, and its training data would predate that. Even after a post-hoc check removed articles containing explicit year references from the generations, the result held: Qwen still assigned an earlier temporal character to GPT2's output.
One example illustrated the effect vividly. On a headline about air quality and mental development in toddlers, GPT2's hidden continuation discussed children with ADHD scoring better on IQ tests. When Qwen received that partial text, it continued in a style recognizably closer to GPT2's voice than to its own natural phrasing, producing details about compass tasks and control groups that tracked GPT2's idiosyncratic narrative structure.
What This Could Mean
The author is careful to call this a hypothesis, not proof. The study is exploratory, and the sample size is small. But if models genuinely learn to approximate the outputs of other LLMs internally, the implications are substantial.
Such internal models would allow a system to simulate its own outputs before generating them, giving it a form of metacognitive preview. It could sample its latent space to estimate uncertainty more accurately. At the outer edge of speculation, it could constitute a kind of self-model, a representation of how the model itself tends to think and write. Notably, a 4B base model already showed these signals; larger models with more capacity would likely encode these representations even more richly.
The study also revealed an important constraint. Instruction-tuned models did not exhibit the same behavior, not because they lack the capacity, but because post-training shifts their objective toward following instructions rather than completing text in an open-ended way. The effect was observed when models were asked to perform the task they were originally trained for: predicting the next token.
The experiment was repeated on a 14B Qwen base model with consistent results. The findings invite replication and extension. What is clear is that the boundary between a model understanding its own output and a model understanding the output of other models may be thinner than assumed.