Language models must continually update their knowledge to remain current. Most existing work focuses on making these post-training updates efficient, but this paper investigates a more foundational question: how the organization of knowledge during pre-training itself shapes how easily new information can be absorbed later.
The authors distinguish two emergent model types. Models trained on chronologically ordered knowledge become specialists—they excel at the most recently encountered time period. Models trained on the same knowledge randomly shuffled become generalists—they perform reasonably across all represented periods. The counterintuitive finding is that specialists require far fewer update steps than generalists to acquire new knowledge.
To explain this advantage, the authors derive a law from first principles linking update cost to the model's optimization dynamics. This law closely predicts the empirical results across their experiments. The derivation connects the model's internal representations to the computational cost of incorporating new knowledge, providing a theoretical backbone for the observed phenomenon.
Guided by these findings, the authors design a pre-training data organization strategy that preserves stable historical knowledge while enabling efficient updates to current knowledge. The benefits extend to real-world pre-trained models and existing knowledge update tools, and the approach generalizes to other knowledge orderings beyond simple chronological versus shuffled splits.
For practitioners, the key insight is that pre-training curriculum matters significantly for subsequent learning efficiency. Deliberately ordering training data by knowledge age or dependency creates models that update more efficiently—a finding that could inform how future foundation models are trained for continual operation.