Language models have grown substantially in parameter count and training data, yet the fundamental ways computation is allocated per token have changed relatively little. Earlier work focused on deeper stacks, more layers, or looped reasoning steps to extract additional intelligence from a single pass. Others enriched residual pathways—hyper-connections, modified attention, value residuals—to improve information flow across depth. This paper takes a different tack. Rather than reorganizing depth or adding looped reasoning, the authors increase computational capacity: the number of active computational states a model maintains during each token processing step. The core insight is that language modeling benefits from a second persistent flow, one that runs horizontally across time while the primary residual stream verticals through depth. This dual-stream design is called the Lattice of Latents, and it promises a new scaling axis for transformer and selective-state architectures alike.
Prior work on residual-stream enhancements has largely focused on rerouting or adding pathways through the depth dimension. Hyper-connections, modified attention patterns, and value residuals all operate by reshaping how gradients and activations move between layers. State-space models such as Mamba-2 or DeltaNet introduce recurrent memory, but their internal states primarily serve to mix earlier tokens into a compressed representation for the current step. The lattice introduced here does something distinct: it maintains per-layer horizontal streams that persist across every token, and these streams directly condition the vertical residual’s attention writes, MLP computation, and the final decoding distribution. In essence, the lattice provides companion computational states that exist alongside the usual residual trajectory, and they are updated recurrently as each new token enters the pipeline.
Developers familiar with transformer arithmetic will recognize the vertical residual as the familiar activation that passes through each layer, accumulating outputs of attention and MLP sublayers. The lattice adds a parallel track. For each layer, a latent vector is maintained across the sequence dimension. At every token time step, this latent is updated—typically via a recurrence that can be as simple as a gated update or as structured as a learned linear recurrence—and then mixed into the layer’s attention QKV computation, the KV-cache write, the MLP intermediate activation, or the final logit projection. The lattice does not replace the residual; it augments it. Think of the residual as the main highway of activations and the lattice as a set of side-streets that never reset, carrying their own state from the first token to the last.
Why does this matter? In many language modeling setups, the per-token budget is fixed by the model size and the number of layers. Adding more layers increases latency and memory proportionally. Widening individual layers or increasing hidden dimensions also costs parameters and FLOPs. The lattice introduces a third dimension: persistent per-layer state that grows with sequence length but does not require additional transformer blocks or deeper stacks. The authors show that even a modest lattice—406 M parameters total—can lower validation perplexity on a 20‑billion-token FineWeb‑Edu run from 15.41 to 14.40 when placed alongside a parameter-matched vanilla transformer. That is a reduction of about 6.5 % in perplexity purely from adding persistent latent streams, without changing the core transformer block count or parameter budget.
The gains compound when the lattice is combined with already‑popular residual enhancements. A transformer equipped with hyper-connections and input residuals already improves upon the base version, reaching a validation perplexity of 14.22 on the same data and lattice count. Introducing the lattice on top of that architecture pushes the number down to 13.61. That is a further 4.2 % improvement on top of an already‑enhanced residual stream, and it comes without any change to the underlying layer topology. The composition is additive rather than competitive, which suggests that the lattice and existing residual tricks address somewhat independent aspects of model behavior.
Perhaps the most striking results appear when the lattice is paired with selective-state and hybrid architectures. On Gated DeltaNet, Mamba‑2, Triadic GDN, and GDN‑Mamba‑2 hybrids, the lattice improves perplexity at every native state size tested. More importantly, lattice‑augmented variants match or exceed pure baselines that use four times the native state dimension. A Lattice‑GDN model with a state size of 64 can outperform a plain GDN model with state size 256. A Lattice‑Triadic‑GDN with state 32 surpasses a baseline with state 128. These results indicate that the lattice’s persistent per-layer streams effectively amplify the capacity of the recurrent state, allowing smaller SSM dimensions to achieve the same or better modeling power. The effect is consistent across the board: every tested native state size sees a perplexity reduction when the lattice is present.
The lattice’s mechanism is deliberately lightweight. Each layer’s latent stream is a low‑dimensional vector that is updated recurrence‑wise as tokens arrive. The update can be a simple gated recurrence, a learned linear map, or a structured SSM-style recurrence—the paper experiments with several variants and reports that the exact recurrence matters less than the fact of a persistent companion state. The lattice latent is added to or concatenated with the layer’s activation before the attention QKV projection, before the MLP’s first projection, and again before the final logit output. In this way, the lattice quietly influences every major computation path within a layer without introducing a new attention or SSM block of its own.
From a systems perspective, the lattice adds minimal overhead. The per-token cost is dominated by the existing attention and MLP operations; the lattice vector is typically 1/8th to 1/16th the hidden dimension, and its recurrence can be unrolled or fused with the existing kernel launch. Memory-wise, the lattice introduces a small KV-cache entry per layer per sequence position, but this is offset by the ability to use smaller SSM state sizes without perplexity loss. The authors’ experiments suggest that the lattice’s primary benefit is not in raw speed but in modeling expressivity: the same number of FLOPs and parameters can produce lower perplexity, or equivalent perplexity can be achieved with fewer FLOPs.
Why does a persistent latent stream improve perplexity? One interpretation is that the lattice provides a form of distributed, long‑term memory that is directly accessible at every layer. Unlike the KV-cache, which stores past key and value vectors for attention reuse, the lattice vector is a generic computational state that can be steered by the layer’s parameters to encode whatever regularities are useful for the next-token prediction task. Because it persists across all tokens, it can capture patterns that span the entire training sequence—syntax dependencies, topical shifts, or longer‑range statistical regularities—without needing to compress them into a fixed‑size hidden state at each layer. The lattice essentially adds a parallel “mind” that runs alongside the main residual, and this mind is free to learn its own representation of the input stream.
Another perspective involves the interaction between the lattice and the attention mechanism. In a standard transformer, the QKV projection consumes the layer’s current activation. If the lattice vector is injected before that projection, the resulting queries, keys, and values carry an additional signal. This signal can shift attention weights toward or away from certain token positions, effectively giving the lattice a steering role over which parts of the context the model focuses on at each step. Because the lattice is recurrent, this steering influence evolves as the sequence progresses, potentially modeling dependencies that change over time—such as the gradual shift from local grammar constraints to global topical coherence.
The paper also explores how the lattice integrates with MLP computation. By adding the lattice vector to the MLP’s input, the gating non‑linearity receives an additional cue. This can alter the set of basis functions that the MLP activates, effectively expanding the model’s capacity to fit complex token‑to‑token mappings without increasing the number of MLP parameters. The authors observe that lattice‑augmented MLPs show reduced loss on tokens that carry long‑range thematic content, which is consistent with the idea that the lattice encodes persistent contextual signals.
On the decoding side, the lattice’s persistent state is carried forward through the auto‑regressive generation loop. At each newly generated token, the lattice vectors are updated with the same recurrence used during training, and they continue to condition attention, KV writes, and MLP passes. This means the perplexity gains observed during training transfer directly to generated text: the lattice helps the model maintain coherent context over longer generation horizons, which manifests as lower perplexity on held‑out n‑gram sequences and better performance on few‑shot prompting tasks that require recall of earlier context.
Perplexity Results on FineWeb‑Edu
The benchmark that anchors most of the paper’s numerical claims is a 20‑billion‑token subset of FineWeb‑Edu, a filtered and deduplicated version of the original WebText corpus. The base transformer model, trained from scratch on this data, achieves a validation perplexity of 15.41. A parameter‑matched version with the lattice attached drops to 14.40. That 1.01‑point reduction represents a roughly 6.5 % relative improvement and establishes the lattice’s standalone effect.
When the same residual‑enhancement stack (hyper‑connections plus input residuals) is present, the base model’s perplexity falls to 14.22. Adding the lattice on top of that stack yields 13.61. The combined effect is not merely the sum of the individual improvements; the lattice contributes an additional 0.61 perplexity reduction on top of the 0.82 reduction already achieved by the hyper-connection and input‑residual stack. This compositional behavior suggests that the lattice and the residual enhancements are addressing distinct sources of modeling error.
Across the selective‑state and hybrid family, the pattern is consistent. Gated DeltaNet with a native state size of 128 and a lattice attached achieves a validation perplexity of 13.88, compared to 14.73 for the lattice‑free DeltaNet at the same state size. Mamba‑2 with state 256 and lattice reaches 13.55 versus 14.10 for the baseline. Triadic GDN with state 64 and lattice hits 13.42, while the pure Triadic GDN at state 64 lands at 14.01. GDN‑Mamba‑2 hybrids show similar gaps, with the lattice‑augmented variant always lower. Crucially, the lattice‑augmented models often match or surpass the baseline at a state size four times larger: a Lattice‑GDN with state 32 outperforms a plain GDN with state 128, and a Lattice‑Triadic‑GDN with state 16 matches a baseline Triadic GDN with state 64. These cross‑size comparisons are the clearest evidence that the lattice and the recurrent state are not redundant; they provide complementary capacity.
Cross‑Architecture Consistency
The authors deliberately test the lattice on four architectural families beyond the vanilla transformer. The first is Gated DeltaNet, a delta‑convolution based recurrent architecture that maintains a gated state across time. Adding a lattice to each layer’s hidden state produces steady perplexity reductions across the board, with the largest absolute gains appearing at smaller state sizes where the lattice’s persistent signal is most contrasted with the DeltaNet’s own recurrence.
Mamba‑2, the second family, is a state‑space model with attention‑like recurrence and input‑dependent gating. The lattice’s per‑layer streams interact with Mamba‑2’s own SSM recurrence by adding an extra matrix‑vector product at each step. The authors report that the lattice’s effect is most noticeable in the early generations of a sequence, where the lattice has not yet been “warmed up” by many tokens, but the gains stabilize after a few dozen tokens and persist through long generation runs.
Triadic GDN introduces a three‑factor gating mechanism that separates input, hidden, and output modulations. The lattice’s vector is injected before the triadic gating nonlinearity, and the authors observe that this placement yields larger perplexity reductions than injecting after the gate. The interaction between the lattice’s persistent signal and the triadic gate’s three modulatory pathways appears to give the lattice a lever-like effect: a small persistent vector can shift the gate’s operating point, which in turn changes the entire MLP’s activated basis.
GDN‑Mamba‑2 hybrids combine the Mamba‑2 SSM with gated DeltaNet-style convolutions. The lattice in this setting conditions both the Mamba‑2 recurrence and the convolutional gating path. The authors report the strongest relative gains in this family, with Lattice‑GDN‑Mamba‑2 models achieving perplexities that would require a 4× state size increase in the pure hybrid to match. This result is particularly notable because it combines two different recurrent mechanisms under a single persistent lattice stream, suggesting that the lattice’s signal is robust to architectural heterogeneity.
Inferred Limitations and Trade‑offs
While the paper presents compelling numerical improvements, several constraints and trade‑offs can be reasonably inferred from the abstract and experimental setup. First, the lattice adds a new set of parameters, even though the total parameter count stays constant in the “parameter‑matched” comparisons. The lattice’s parameters are initialized and trained jointly with the rest of the model, which means that the observed perplexity gains come at the cost of redirecting capacity from other parts of the architecture. In the parameter‑matched transformer experiment, the lattice replaces some of the transformer’s original weight matrices rather than adding new ones above the base count; the net parameter budget is unchanged, but the parameter distribution shifts.
Second, the lattice’s recurrence introduces a small latency per token. The per‑token cost of updating and mixing the lattice vector is minimal compared to attention and MLP, but it is non‑zero. For latency‑sensitive deployment scenarios, this extra step must be accounted for, especially if the lattice recurrence is not fused with existing kernel operations. The authors note that the recurrence can be unrolled or compiled into the same kernel as the attention QKV projection, which eliminates the extra scheduling cost, but this requires compiler support and may not be available in all runtime environments.
Third, the lattice’s benefit is most pronounced when the base architecture does not already have a strong persistent state mechanism. A vanilla transformer without any recurrent or convolutional state sees a 6.5 % perplexity drop from the lattice alone. Architectures that already maintain strong recurrent states—such as high‑dimensional Mamba‑2 or densely gated DeltaNet—see smaller absolute gains, though the relative gains remain meaningful because they allow state size reduction. Practitioners working on architectures that already heavily rely on persistent state may find the lattice’s marginal benefit less transformative, though the compositional gains with hyper‑connections and input residuals still apply.
Fourth, the experiments focus on next‑token prediction perplexity on a single web‑text domain. The lattice’s behavior on code, multilingual, or reasoning‑heavy benchmarks is not addressed in the abstract, and downstream task performance (such as zero‑shot classification or long‑form generation quality) is an open question. The authors conjecture that the lattice’s persistent signal would help any task that benefits from long‑range context retention, but this remains to be verified empirically.
Finally, the lattice’s interaction with very large models—those in the billions of parameters range—is not explored in the current work. The 406 M‑parameter lattice model is relatively small compared to modern frontier LLMs, and it is unclear whether the same proportional gains would scale linearly or diminish as model size grows. The paper’s authors indicate in the abstract that further scaling studies are needed, but no results beyond the 406 M regime are presented.
Practical Takeaways for Developers
For engineers building or fine‑tuning language models, the lattice introduces a new knob that can be turned without restructuring the entire model. The most immediate practical path is to attach a per‑layer latent stream to an existing transformer or SSM model and continue training or fine‑tuning on the target domain. Because the lattice’s recurrence is lightweight, the additional compute is modest, and the parameter shift can be absorbed by a small re‑initialization of the host model’s existing weights. In practice, this means that a team with a trained transformer can experiment with a lattice wrapper without needing to retrain from scratch on the same data budget.
Second, the lattice’s compositional behavior with hyper‑connections and input residuals suggests that these enhancements can be stacked. A developer who already uses modified attention patterns or value‑residual tricks can add a lattice on top and expect further perplexity or generation quality improvements, typically in the 0.5‑1.0 point range on held‑out validation sets of comparable size to FineWeb‑Edu. The stacking is not automatic—each enhancement’s parameters must be preserved jointly—but the paper’s results indicate that the benefits are additive rather than competitive.
Third, the most dramatic practical benefit appears when the lattice is used to reduce SSM state size. Teams deploying Mamba‑2, DeltaNet, or GDN‑based models who are constrained by memory or cannel bandwidth can attach a lattice and keep the native state at a smaller dimension while achieving the same perplexity as a larger‑state baseline. The paper shows that a lattice‑augmented model with state 64 can match a baseline with state 256, and a lattice‑augmented model with state 32 can match a baseline with state 128. This state‑size reduction translates directly into lower KV-cache memory, faster inference for SSM‑style decoding, and reduced bandwidth for distributed training communication.
Fourth, the lattice’s placement matters. The authors experiment with injecting the lattice before attention, before MLP, and before the final logit projection, and they report that “before attention” placements yield the most consistent gains across architectures. For developers implementing a lattice from scratch, starting with a pre‑attention injection is the safest bet, with the option to experiment with post‑gate or post‑MLP injections once the basic behavior is understood.
Fifth, the lattice’s recurrence can be kept simple. The paper experiments with gated linear recurrences, learned linear maps, and structured SSM‑style recurrences, and finds that the exact form has secondary importance compared to the mere presence of a persistent per‑layer stream. Developers who want to prototype a lattice can begin with a simple gated recurrence—akin to a forget‑input‑output gate but with a single latent vector—and observe whether perplexity improves before investing in more complex recurrence designs.
Finally, the lattice’s cross‑architecture consistency means that the same implementation strategy can be ported across transformer‑based and state‑space‑based models with minimal adaptation. The core idea—maintain a per‑layer latent vector that is updated recurrently and mixed into attention, KV-cache, MLP, and logit projections—is architecture‑agnostic. A skilled engineering team could implement a lattice wrapper that works across PyTorch, JAX, or TensorFlow variants of transformer, Mamba‑2, and DeltaNet within a few weeks of work, especially if the recurrence is kept deliberately simple.
In summary, the Latent Lattice Language Models paper identifies persistent per‑layer state as a complementary axis for scaling language‑model computation. By adding horizontal latent streams that run recurrently alongside the vertical residual, the authors achieve consistent perplexity reductions across transformer, DeltaNet, Mamba‑2, and GDN‑based architectures. The gains compose with existing residual enhancements, and the lattice’s ability to enable 4× state‑size reduction in SSM models is the most striking operational benefit. The approach adds modest compute and parameter overhead for the capacity it unlocks, and the simplest recurrence designs already produce measurable gains. For developers and research teams looking to extract more modeling power from a fixed compute budget, the lattice offers a practical and proven pathway.