Three and a half months ago, a developer published a post about building a vintage language model from scratch, trained exclusively on texts written before 1900. The initial attempt produced a 340 million parameter model that could generate Victorian paragraphs but could not hold a conversation. Since then, the project has consumed every single day of work, resulting in new datasets, new models, a custom benchmark, and an evaluation pipeline. The updated results show what happens when someone treats data quality as a first-class engineering problem.
The base model problem was data, not architecture
The original model was generating broken sentences, page numbers, and OCR noise. The developer could not initially tell how bad it was because there was nothing to compare against. Running plain completions on other people's vintage base models, TypeWriter, Talkie, and various Llama variants, produced proper prose without any fine-tuning. The original model did not.
The diagnosis was straightforward: the base training data was insufficient. But "insufficient" could mean not enough data, or data of insufficient quality. The answer turned out to be both, and the developer spent months pursuing both paths simultaneously.
Four historical book collections expanded the training corpus
The developer found four collections of transcribed books that had been largely overlooked: CLMET 3.1 with 335 late modern English books, ECCO with 3,102 books from the 1700s, EEBO with 34,937 books spanning 1470 to 1700, and EVANS with 5,013 early American books. These were significantly cleaner than the OCR newspapers used in the original dataset, though they arrived in a format that required substantial conversion work to produce clean Markdown.
These collections were merged with the original dataset, Hayk's 1800-1875 corpus, and jbduran's Institutional Books dataset into a combined set called Sprocket-n-Say: 15 million rows totaling 16.7 billion tokens. A larger version with more relaxed quality filters, additional books from Gutenberg, Archive.org, and Authorama, plus synthetic data, reached 49 million rows and 40 billion tokens under the name Piston-n-Prose.
Synthetic data became necessary despite initial resistance
The original project rejected synthetic data on principle. The whole point was to train on real texts from real people who lived before 1900. But OCR garbage proved to be a fundamental obstacle. Broken words, hyphens at line endings, random symbols, and paragraphs transcribed three times slightly differently made it difficult for the model to learn coherent English.
The developer tried fine-tuning the base model repeatedly and found that it did not improve meaningfully. The realization was that you cannot fine-tune knowledge into a model that never learned it in the first place. An undertrained base model stays undertrained regardless of how many conversation examples you show it. Fine-tuning teaches it to be confidently wrong in a nicer format. The fix had to come from the base-training data.
The synthetic approach borrowed from HuggingFace's Fine-Phrase paper used for SmolLM models. High-quality texts were chunked into approximately 4,000 word segments and processed through Gemma-4-E4B running locally with vLLM. The model rewrote each chunk in several ways: continuing it as a lost manuscript, extracting facts as educational text, converting it to a magazine article, or retelling it as a story. The goal was to preserve vocabulary and knowledge from the original texts while removing OCR artifacts.
A second synthetic dataset took a different approach. Two vintage models, TypeWriter-7B and Talkie-13B, were given tiny two to three word seeds extracted from the datasets and allowed to complete the text however they wanted. Millions of completions were generated, then filtered to keep the best output. The result was effectively asking old models to ramble about random topics continuously, producing text that preserved the vintage style without contamination from modern content.
A word census revealed hidden modern contamination
While cleaning the data, the developer counted every word across 41 billion words of English written before 1900. For each word, the count was tracked per source and in total, along with pairs and triplets. Any word appearing zero times in the full corpus was flagged as potentially modern.
The first version revealed a surprise: "internet" appeared 7,175 times in pre-1900 text. Investigation found the source. Scanned books contained "Digitized by the Internet Archive in 20xx" printed at the top. Gutenberg books included proofreader email addresses and references to the Online Distributed Proofreading Team. The scanning metadata was leaking into the word counts.
More significantly, many words that appeared modern were actually vintage, just used in different contexts. A computer in 1850 was a person who computes, a clerk doing sums all day. Google eyes were bulging eyes. A tweet was the note of a small bird. A robot was forced labor that peasants owed to their landlord in Eastern Europe. Digital referred to digitalis, the foxglove heart medicine. Radio was Latin for rays, the spokes of a wheel. Smog was the OCR corruption of cosmogony. A lesbian was simply a person from the island of Lesbos. Queer meant strange or odd, and many instances were bad OCR readings of "Queen of Heaven." A faggot was a bundle of sticks, and "fire and faggot" described burning a heretic.
Each suspicious word had to be examined in context individually before deciding whether to ban it. The banned word list eventually reached approximately 800 entries, but the process of building it taught a lesson about the gap between assumptions about historical language and actual historical usage.
Fine-tuning data required a different approach
Base training is about quantity. Fine-tuning is about teaching the model to hold a conversation, which is harder when no Victorian chat logs exist. The developer tried three approaches.
Genuine data came from 11,218 exam questions and answers extracted from six school exam books published between 1883 and 1900. These were real questions that real students answered, grounded in actual Victorian knowledge. Semi-synthetic data chunked pre-1900 books and asked different LLMs to generate questions and answers from each chunk, keeping the output grounded in the original style and ideas. Fully synthetic data started with hundreds of seed questions a person might have asked in 1890, like "What are the duties of a gatekeeper at a railway crossing?" and "Tell me a recipe for making a savoury apple dumpling." These seeds generated similar questions and answers through multiple LLMs, all filtered against the banned word list to prevent modern content from leaking in.
The largest fine-tuning dataset, Patina-1, was created by taking 310,000 rows from popular modern datasets including OpenHermes, smoltalk, UltraChat, and SlimOrca, and rewriting them through a Victorian style-transfer LoRA called Patina trained on Qwen3-8B. This preserved the conversational structure of modern datasets while converting the language to a vintage register.
The book destruction problem
The developer included a section addressing the practice of AI companies destroying physical books to scan them for training data. The argument is straightforward: companies like OpenAI, Anthropic, and Amazon are cutting spines, scanning pages, and discarding the physical copies. Some of these are rare books that cannot be replaced. The contribution of any single book to a training dataset of trillions of tokens is negligible, yet the physical destruction is permanent.
The developer pointed out the contradiction of years of consumer messaging about digitizing everything to save trees, followed by the destruction of printed books for AI training. This is a tangent from the technical work, but it reflects a concern that the AI industry's data practices are not limited to digital sources and carry real consequences for physical cultural artifacts.
What the project demonstrates
The vintage LLM project is a case study in what it takes to build a domain-specific language model from scratch. The developer built custom datasets, created synthetic training data, built a word census to identify contamination, constructed a benchmark and evaluation pipeline, and trained multiple model iterations. The code, datasets, and models are all published openly.
The practical lesson is about data. The difference between a model that generates broken OCR noise and one that produces coherent Victorian prose is not architecture or compute. It is the quality and composition of the training data. The developer spent more time on data than on anything else, and the results justify the allocation. The models are available on HuggingFace, the code is on GitHub, and a demo space lets anyone interact with the results directly.