A memory that reproduces every current prediction may still have forgotten how to learn from the next observation. We characterize this distinction for exact cumulative learning and attention. For smooth convex residual losses with unrestricted real features and labels, a continuous recurrent state of fixed dimension exists exactly when the loss is polynomial. With ordinary, unweighted observations, degree $D$ requires precisely $r=\binom{d+D}{D}-1$ coordinates; even among histories with one fixed current predictor, exactly $r-d$ coordinates can remain necessary for continuation. Nonpolynomial losses have unbounded requirements on such equal-prediction histories. Bounded scalar logistic examples retain this obstruction after fixing any finite derivative jet. For ridge regression, the hidden evidence is the design matrix: the exact state costs $dp+d(d+1)/2$, versus $dp$ for the predictor. On a well-conditioned zero-prediction family, optimal $B$-bit compression has average one-update error of order $d^{-1}2^{-2B/s_d}$, $s_d=d(d+1)/2$, even with arbitrary randomized codes. Kernel attention gives a second exact closure classification. The results identify what future updates reveal, rather than proposing a new regression algorithm or asserting trained-model gains.
Memory vs. Learnability in Recurrent States
The paper establishes a fundamental distinction: a memory that perfectly reproduces current predictions may nevertheless be insufficient for learning from the next observation. This separation between reproduction and learnability is characterized across exact cumulative learning and attention mechanisms, with precise conditions on when a fixed-dimensional recurrent state can capture all necessary information.
Polynomial Losses and Fixed-Dimensional States
For smooth convex residual losses with unrestricted real features and labels, a continuous recurrent state of fixed dimension exists exactly when the loss is polynomial. This result provides a clean characterization: polynomial losses admit a fixed-dimensional state, while nonpolynomial losses do not. The implication is that the algebraic structure of the loss function dictates whether a compact memory representation is possible.
Coordinate Complexity for Polynomial Losses
With ordinary, unweighted observations, degree $D$ requires precisely $r=\binom{d+D}{D}-1$ coordinates. Even among histories with one fixed current predictor, exactly $r-d$ coordinates can remain necessary for continuation. This combinatorial count reveals the precise dimensionality needed to encode sufficient statistics for polynomial losses of a given degree, and identifies which coordinates remain essential beyond the current predictor.
Obstructions for Nonpolynomial Losses
Nonpolynomial losses have unbounded requirements on equal-prediction histories. Bounded scalar logistic examples retain this obstruction after fixing any finite derivative jet. This means that no finite set of derivatives can resolve the learnability gap for nonpolynomial losses; the memory requirements grow without bound as the model seeks to learn from new observations.
Ridge Regression and the Design Matrix
For ridge regression, the hidden evidence is the design matrix: the exact state costs $dp+d(d+1)/2$, versus $dp$ for the predictor alone. The additional $d(d+1)/2$ terms capture the accumulated second-order information in the design matrix that the predictor summary discards. This gap quantifies exactly what is lost when compressing the state to just the current prediction.
Compression Error Under Limited Bit Budget
On a well-conditioned zero-prediction family, optimal $B$-bit compression has average one-update error of order $d^{-1}2^{-2B/s_d}$, $s_d=d(d+1)/2$, even with arbitrary randomized codes. The $s_d=d(d+1)/2$ term is the same design-matrix dimension from ridge regression, confirming that the compression bottleneck stems from the same second-order information. The error decays inversely with dimension and exponentially with the bit budget per relevant coordinate.
Kernel Attention and a Second Closure Classification
Kernel attention gives a second exact closure classification. Just as polynomial losses admit fixed-dimensional recurrent states, kernel attention mechanisms have their own structural condition for when the attention state can be represented in fixed dimension. This provides a parallel classification for attention-based memory beyond the residual-loss framework.
What the Results Imply for Memory Design
The paper does not propose a new regression algorithm nor assert gains from trained models. Instead, it identifies what future updates reveal about the information that must be retained. For system designers, the key takeaway is that the loss function and observation structure dictate the minimum memory dimensionality required for learnability. Designing recurrent states without accounting for these conditions risks creating memories that reproduce predictions but fail to support subsequent learning.
As agents and models grow increasingly reliant on recurrent memory for online adaptation, understanding the precise information requirements becomes essential. Future work might explore how these theoretical bounds translate to practical architecture choices, or whether approximate memory representations can approach the identified limits. Read the paper on arXiv