LLM-generated stories show very low variability across current models. A study sampling 20,000 total stories from four current models using five prompts found that 11 words occur in 88.3% of generated stories, with little difference between models. These words include names such as Elias, Mara, Elara, settings such as lighthouses, and professions such as clockmaker, librarian.
Recurring tokens appear outside typical training sources
These tokens do not often occur in published literature nor pre-training data. They are found instead in preference data that is likely to have been used by all current models. The dominance of these specific tokens suggests that alignment objectives shaped through preference modeling outweigh broader sources in determining story content.
Lighthouse stories are infrequent relative to typical output
Surprisingly, the "lighthouse" stories featuring these recurring elements are infrequent when compared with the average post-training story. Much of the average post-training story contains references to copyrighted characters or adult content. This contrast indicates that the preference data driving alignment has a disproportionate impact relative to the diversity present in the broader training distribution.
Small datasets combined with powerful alignment algorithms
The result demonstrates the potentially disproportionate impact of small datasets combined with powerful alignment algorithms. When alignment objectives concentrate on a limited set of preferred tokens or themes, the resulting output exhibits low variability even across models with different underlying architectures and training histories.