When Running Out of Data Breaks Sparse Models

The internet's text is finite. As language model training scales toward and beyond the limits of human-written corpora, repeating data has become standard practice. Muennighoff et al. (2023) showed that dense Transformers tolerate moderate repetition, roughly four epochs over a fixed corpus, before held-out loss degrades. But the models that actually dominate production today are not dense. Mixture-of-Experts architectures route each token to a small subset of many parallel feed-forward networks, achieving the same FLOPS as a much smaller dense model while maintaining a far larger parameter count. The interaction between this sparsity and data repetition has been almost entirely unexplored.

This paper from Atindra Jha, Margaret Li, Jure Leskovec, Percy Liang, and Luke Zettlemoyer at Stanford and the University of Washington addresses that gap with a systematic grid sweep. They train dense and MoE Transformers at three active-parameter scales, 80M, 200M, and 1B active parameters (up to 8.5B total), varying expert count, expert granularity, repetition rates, data domains, data mixes, and quality filters. The central finding is stark: MoEs degrade more rapidly and more catastrophically under data repetition than dense models, and the mechanism is tied to the total parameter count rather than the active parameter count.

How MoEs and Data Repetition Interact

In a Mixture-of-Experts Transformer, the feed-forward network at each layer is replaced by n parallel expert FFNs and a router that selects the top-k experts for each token. The sparsity ratio is n/k: a model with 64 experts and top-4 activation has a sparsity of 16, meaning only 1/16th of the FFN parameters are active for any given token. This decouples total parameters from active parameters, and it is this decoupling that creates the problem.

The authors fix the total training token budget T at 20 times the active parameter count, following Chinchilla-style scaling, and vary the repetition rate R by changing the number of unique tokens U = T/R. They consider R in {1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024}, training on subsets of a fixed random permutation of the OLMoE data mix to ensure nested training sets.

The results are consistent across all three scales. Dense 80M models tolerate repetition up to about 8x with minimal degradation, and only show sharp loss increases beyond R=64. MoE models at the same active parameter count begin suffering at R=4 and deteriorate rapidly. At R=32, MoEs cede their performance advantage and begin to underperform dense models. At R=64 and beyond, MoEs dramatically underperform their dense counterparts.

Total Parameters Drive the Damage

The most surprising finding is that overfitting under repetition depends on total parameters, not active parameters. The authors vary expert count (n in {8, 16, 32, 64, 128}) and expert granularity (g in {1/2, 1/4, 1/8, 1/16, 1/32}), keeping active parameters fixed. In both cases, more total parameters produce sharper degradation under repetition. An MoE with 128 experts of granularity 1/4 (total parameters roughly 4x the active count) overfits much more than an MoE with 16 experts of the same granularity.

To confirm this, the authors compare a 200M dense model (total parameters = 193.9M) against 80M active MoE models with 158M total parameters (MoE 32x1/4) and 244M total parameters (MoE 64x1/4). The 200M dense model's repetition response falls between the two MoE models, tracking total parameters rather than active parameters. This makes intuitive sense: each expert sees only tokens routed to it, and with more total experts, each expert's effective training set is smaller. Repeating data means each expert repeatedly trains on the same narrow shard.

Importantly, the effect does not depend on the total data budget. Training with 4x more tokens (T/N_a = 80 instead of 20) produces nearly identical repetition curves, confirming that the ratio of unique tokens to total parameters, not total tokens, is what matters.

Robustness Across Domains and Filters

The authors test four distinct data domains: DCLM (web crawl), StarCoder (code), peS2o (academic text), and Wikipedia (encyclopedic text). All four exhibit the same pattern: MoEs overfit more than dense models, and the crossover point where MoEs lose their advantage falls in the R=16 to R=32 range. The semantic diversity of these domains, from structured code to academic prose, does not change the fundamental dynamics.

Data quality filtering has only a modest effect. The authors interpolate between DCLM-pool (unfiltered, 280T tokens) and DCLM-baseline (heavily filtered, retaining 2.4% of the pool). Training on higher proportions of unfiltered data results in consistently worse performance, but the MoE-vs-dense overfitting gap persists regardless of the filter mixture. Notably, at the extreme of R=1 on all-unfiltered data versus R=32 on all-filtered data, the unfiltered single-pass wins, suggesting that raw quantity beats filtered repetition, though an intermediate combination is likely optimal.

Mixed Repetition Rates in Data Mixes

In practice, language models train on data mixes where different domains are repeated at different rates. GPT-3, for example, repeated Wikipedia 3.4 times but used Common Crawl less than once. The authors study this by mixing a non-repeated domain (DCLM) with a repeated domain (peS2o or StarCoder) at various proportions.

For StarCoder mixed with DCLM, code validation loss rises according to the same pattern as single-domain experiments regardless of the StarCoder mixing percentage. Web crawl loss is unaffected unless StarCoder occupies at least half of the total budget. For peS2o mixed with DCLM, the pattern is different: the unrepeated DCLM component appears to regularize the repeated peS2o, dampening the degradation at high repetition rates. The authors attribute this to the higher semantic similarity between academic text and web text compared to code and web text, suggesting that mixing repeated data with unrepeated data from a semantically similar domain may be a viable mitigation strategy.

What Works and What Doesn't as Remedies

The authors evaluate seven regularization methods: dropout, gradient norm clipping, weight decay, FFN output masking, expert dropout, expert output masking, and router jitter.

Three methods have no measurable effect. Gradient norm clipping across thresholds {0.2, 1.0, 2.0} produces results within noise. Weight decay sweeps over {0.05, 0.1, 0.2, 0.4} show small, well-ordered differences that remain within variance. Router jitter, which multiplies the router's input by uniform noise on [1-epsilon, 1+epsilon] during training, shows no clear impact across epsilon in {0.0, 0.1, 0.2, 0.4}. These three methods operate by reducing update strength, shrinking weight magnitude, or perturbing coarse gradient paths, none of which address the core memorization mechanism.

Four methods work. Standard residual dropout at probability p=0.4 dramatically reduces the impact of extreme data repetition for all architectures, though it hurts performance at low repetition rates. FFN output masking, which zeroes the entire FFN or MoE output for a token with probability p, incurs a smaller penalty at low repetition while providing similar regularization at high repetition. Expert dropout, applied only to expert hidden activations, behaves similarly to FFN output masking. Expert output masking, which independently zeroes each (token, expert) output, shows comparable effects.

With strong masking-based regularization, specifically dropout at p=0.4, MoEs are able to outperform dense models even at repetition rates above 64x. However, no method fully matches the performance of all-unique training data. The gap between regularized repeated-data training and single-pass unique-data training persists across all settings tested.

The Router Freezes Early, and Experts Over-Specialize

The mechanistic investigation reveals why MoEs are more vulnerable. The authors measure routing stability by recording each token's top-1 expert at every MoE layer across training checkpoints, then computing the fraction of tokens whose top-1 expert did not change between consecutive checkpoints.

Routing ossifies rapidly. At the first checkpoint (step 200), routing is essentially random, with stability near 1/n for n experts. By step 400, just 10% of training, stability reaches 60%. By the end of training, stability exceeds 95% for all configurations. This means that after early training, each expert's token set is nearly fixed. Under data repetition, each expert repeatedly sees the same narrow shard of tokens, rather than the full data distribution.

Higher repetition exacerbates ossification. End-of-training routing stability rises with repetition rate across all model scales and MoE configurations. However, at extremely high repetition (R=32 and above), the router appears to destabilize, showing lower stability than at intermediate rates. This may reflect a phase transition where the router can no longer maintain meaningful routing structure.

Expert specialization, measured by the increase in cross-entropy loss when a single expert's output is zeroed (expert knockout), rises with data repetition. At 80M active parameters, increasing R from 1 to 32 increases the knockout effect by 1.1x for 16 experts and by 2.3x for 128 experts. More experts mean each expert is more specialized and less redundant, so removing one has a larger impact. This pattern mirrors the main overfitting result: more total experts means more overfitting under repetition.

Expert co-activation entropy, which measures how uniformly expert pairs are co-activated for the same token, rises steadily with repetition toward the uniform distribution. This means that at high repetition, the routing decisions become more diffuse even as individual experts become more specialized, a seemingly contradictory pattern that reflects the router's attempt to distribute memorized tokens across the expert pool.

Dropout's regularizing effect does not operate through router plasticity. Models with dropout show less routing ossification at R=64 than models without dropout, but dropout also partially recovers performance without changing the fundamental routing dynamics. The authors hypothesize that overfitting is driven not by routing itself, but by the functions learned within each expert. Dropout combats this by reducing dependence on any single expert and enforcing multiple, more varied feature representations.

Implications for MoE Training Under Data Constraints

The practical takeaway is unambiguous: the compute efficiency advantage of MoEs is conditional on having enough unique data. If you are training on a data-constrained budget where repetition is necessary, dense models are more robust. The crossover point where MoEs lose their advantage is around 16-32x repetition for models in the 80M to 1B active parameter range.

For teams training MoEs on data-constrained budgets, the findings suggest several strategies. First, strong dropout (p=0.4 or higher) or FFN output masking can extend the useful range of MoEs under repetition, allowing them to outperform dense models even at 64x repetition. Second, mixing repeated domains with unrepeated data from semantically similar sources may provide additional regularization. Third, reducing the total parameter count by using fewer, larger experts may reduce vulnerability to repetition, since the overfitting effect scales with total parameters.

The mechanistic findings also point toward future research directions. If expert over-specialization is the core problem, methods that explicitly disrupt memorization patterns within experts, such as stochastic expert pruning, adaptive routing that reshuffles token assignments during training, or contrastive objectives that encourage experts to learn complementary rather than specialized representations, could provide more targeted remedies than generic regularization.