Latent neural surrogate solvers promise to accelerate simulations of physical systems by orders of magnitude — evolving a compressed latent representation rather than resolving full-resolution fields at every timestep. In principle, this is a win: less computation, faster predictions, and the ability to use memory-intensive architectures like transformers that would be impractical at full resolution. In practice, however, these models suffer from a persistent problem: errors accumulate rapidly during long autoregressive rollouts, making predictions unreliable beyond a short horizon. A team led by Rémi Dingreville at Sandia National Laboratories, alongside collaborators from the University of Southern California, Brown University, and Oak Ridge National Laboratory, has identified the root cause of this instability and systematically addressed it through training-level interventions.
The Root Cause Is Not the Latent Space
The prevailing assumption has been that rollout instability is an inherent limitation of compressing complex physical dynamics into a low-dimensional representation. The paper challenges this assumption directly. The instability, the authors show, stems from how the latent representation is trained, not from the representation itself. In the standard sequential training paradigm, the autoencoder is optimized purely for reconstruction accuracy — it learns to faithfully reproduce its input — and only afterward is a separate dynamics model trained to predict trajectories within the resulting latent space. Because the autoencoder never observes the rollout task during its initial training, the representation it produces can be highly effective for reconstruction yet poorly suited for long-horizon forecasting.
This is analogous to the shift that has been underway in generative modeling and computer vision. Vector-quantized autoencoders (VQGANs) and latent diffusion models are no longer trained solely to reconstruct images; they are trained to produce representations that are useful for downstream tasks like image generation or denoising. The paper applies this same insight to scientific computing: the compression mechanism should be trained to restructure the solution space for stable dynamical evolution, not merely to minimize pixel-wise reconstruction error.
Five Training-Level Interventions
The team systematically evaluated five interventions applied during the autoencoder training phase:
Spatially structured latent encodings. Conventional LDMs compress each field into a flat vector through a convolutional autoencoder with a small fully connected bottleneck, reaching compression ratios of up to 500:1. Here, the team adopted a fully convolutional wavelet-based autoencoder that retains spatial structure in the latent encoding, compressing far less aggressively at a ratio of 64:1. This is the only architectural change considered; all other interventions preserve the baseline architecture.
Koopman-inspired dynamics constraints. During autoencoder training, a second reconstruction term penalizes the reconstruction of the subsequent time step, encouraging the latent space to support approximately linear dynamics. A learned Koopman operator is used during training but discarded afterward, replaced by the dynamics model. This constraint is applied with only weak penalization relative to the primary reconstruction loss to encourage linear structure without degrading reconstruction quality.
KL divergence regularization. A Kullback-Leibler penalty encourages the latent encoding to adopt a unit Gaussian structure, constraining the latent space from expanding arbitrarily and helping maintain a well-behaved geometry for the dynamics model to operate in.
Noise injection. This intervention operates at two stages. Latent-space noise injection — called Hamming noise injection — adds constant-variance white Gaussian noise directly into the latent encoding after the autoencoder compresses the field but before the Koopman transformation. This forces the autoencoder to separate nearby latent points, learning representations that are robust to corruption and where the dynamics model only needs to target a small region around each encoded point. Dynamics-side noise injection adds white noise to the dynamics model's input during pretraining, exposing it to perturbed latent trajectories.
Recursive (multistep) training. After pretraining the dynamics model, it is fine-tuned by rolling out its own predictions for additional steps before computing the loss, but only on the final output. This exposes the dynamics model to its own accumulated prediction errors during training, forcing it to learn to correct for its own mistakes rather than assuming access to ground-truth latent context at every step.
The Surprising Trade-off: Better Rollout, Worse One-Step Accuracy
One of the most important findings is that interventions that improve long-horizon rollout stability often degrade conventional training metrics, including reconstruction accuracy and one-step prediction accuracy. This is not a flaw in the approach but a direct consequence of misalignment between standard training tasks and the actual objective of neural solvers. A latent representation that perfectly reconstructs its input at every timestep may encode features that are irrelevant or even destabilizing for long-horizon evolution. Sacrificing some reconstruction fidelity can produce a representation whose geometry is better suited to stable dynamical propagation.
This finding has broad implications for how scientific machine learning models are evaluated. Conventionally, reconstruction loss or one-step prediction accuracy serves as the primary metric. The paper argues that these metrics are insufficient proxies for long-horizon predictive utility, and that optimizing for them can actively harm the property that matters most: stable, accurate rollout over many steps.
Results Across Three Physics Problems
The team evaluated their approach across three fundamentally different physical systems, each stressing a different aspect of latent dynamics prediction.
Spinodal decomposition describes phase separation in an initially homogeneous mixture, modeled by the Cahn-Hilliard equation. This single-field problem with comparatively slow dynamics serves as a controlled baseline, allowing verification that the interventions recover performance previously attributed only to pixel-space models.
Active matter models a dense suspension of active particles in an incompressible Stokes fluid, coupling eleven simultaneously evolving fields including particle concentration, fluid velocity, and orientation and strain-rate tensors. The problem contains an implicit closure issue — the evolution of resolved orientational moments depends on unresolved higher-order moments — making it substantially more demanding than spinodal decomposition. This became the primary testbed for systematically studying each intervention.
High-cycle fatigue predicts damage accumulation in polycrystalline metals from crystal-plasticity simulations. This is the most challenging case: the phenomena of interest occurs on exponential time scales while conventional numerical solvers operate on linear time scales to retain stability. Training data is logarithmically subsampled, limited to simulations of 2^9 cycles or fewer, yet the model must extrapolate to cycle counts far beyond its training data. The dataset resolves 55 co-evolving field quantities on a 128×128 grid.
Collectively, the interventions reduce long-rollout error by approximately 40% on the active matter problem and match or exceed the accuracy of full-resolution models on both the spinodal decomposition and active matter benchmarks. The resulting models require two orders of magnitude fewer floating point operations and half the GPU memory of comparable full-resolution solvers.
Extrapolation Beyond Training Horizons
The most striking demonstration of the approach's value is in high-cycle fatigue prediction. The model is trained only on computationally accessible cycle counts (up to 2^9 cycles) but must extrapolate to horizons orders of magnitude beyond that. Using a relative-time encoding that allows rollout at cycle counts outside the training distribution, the resulting surrogate achieves stable extrapolation far beyond its training horizon. This is not merely a matter of stable rollout over a fixed number of steps — it is the ability to predict physical behavior at time scales that were never observed during training, a capability that is essential for practical engineering applications where long-term degradation must be forecasted.
The paper also explores the applicability of the Walrus foundation model, a 1.3-billion parameter transformer pretrained across nineteen physical scenarios spanning fluid dynamics, acoustics, and astrophysics, as an alternative dynamics model operating within the compressed latent space. The spatially structured latent encodings make this compatibility possible, as features learned during physical pretraining in real space may remain relevant when applied within the compressed space, provided that space retains enough physical structure.
What This Means for Scientific Computing
The paper's central message reframes how we think about neural compression in scientific applications. Compression is not merely a dimensionality reduction exercise — it is a restructuring of the solution space. The geometry of the latent representation determines whether the dynamics model can reliably propagate predictions forward in time, and this geometry should be shaped by the requirements of long-horizon evolution rather than by the demands of reconstruction.
For practitioners, the practical implications are clear. Neural surrogates that require 100 times less computation and half the GPU memory of full-resolution models, while matching or exceeding their accuracy, make previously prohibitive simulations tractable. The training-level interventions are straightforward to implement — they modify how an autoencoder is trained, not its architecture — and can be applied to existing sequentially trained LDMs without fundamental redesign.
The limitations are also worth noting. The paper focuses on three specific physical systems, and the generality of the findings across other problem types remains to be established. The diagnostic metrics introduced — decoder sensitivity and latent space velocity — provide insight into why interventions work but add computational overhead. And while the high-cycle fatigue results are promising, they demonstrate extrapolation within a single problem class; broader validation across diverse physical systems would strengthen confidence in the approach's generality.
More broadly, the paper establishes that designing neural compression for stable dynamical evolution is not an optional refinement but a key requirement for building reliable, efficient neural surrogates for scientific applications. The path forward is not simply to build bigger or more expressive models, but to train the compression mechanism with the real objective in mind: stable, accurate prediction far into the future.