Sparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. We propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding. This is motivated by our synthetic analysis showing that coefficients from a single linear encoder pass are suboptimal compared with those obtained by per-sample optimization, suggesting that sparse coding is a reliable mechanism for dictionary learning. Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery. These gains translate to state-of-the-art cross-seed consistency across four vision and language settings, where ConSDL improves dictionary and coefficient consistency by 17.6% and 35.9%, respectively, over the average of six SAE baselines.
The Reproducibility Crisis in Sparse Autoencoders
Sparse autoencoders (SAEs) have become a widely used approach for analyzing the internal representations of foundation models. SAEs decompose each model activation into a sparse linear combination of vectors from a learned overcomplete dictionary. Empirically, these dictionary vectors often align with human-interpretable concepts, such as colors, textures, and objects in vision models or characters, words, and syntactic patterns in language models. Importantly, these concepts can be learned without a predefined vocabulary, enabling scalable explanations of model behavior in vision and language, audits of hidden biases in models and datasets, and identification of candidate factors in scientific and medical data.
Yet a critical limitation remains before SAEs can serve as practical tools for analysis. SAEs trained on the same data with the same configuration can learn substantially different dictionaries across random seeds. This limits reproducibility and reliability. More fundamentally, it raises the question of whether the features learned by SAEs genuinely reflect the model's internal representations or are artifacts of the training procedure. Recent work has further heightened this concern by showing that SAEs can fail to recover known ground-truth features in controlled synthetic settings.
From Amortized Inference to Per-Sample Sparse Coding
Standard SAEs use a linear encoder followed by a non-linearity to predict the sparse coefficients from each input in a single forward pass. These coefficients directly influence the gradients used to update each dictionary atom during training, yet coefficients produced by a single encoder pass can be suboptimal compared with those optimized for the current dictionary. We therefore replace the encoder in a conventional SAE with per-sample sparse coding and infer the coefficients for each input as the solution to an optimization problem that minimizes reconstruction error under an l1 penalty, controlling sparsity with a parameter lambda greater than zero. We estimate the solution using the Fast Iterative Shrinkage-Thresholding Algorithm. Each FISTA iteration updates the coefficients by taking a gradient step on the reconstruction term, followed by nonnegative soft thresholding for the l1 penalty.
Coefficient Refitting: Correcting Shrinkage Without Activating New Atoms
While the l1 penalty promotes sparsity by penalizing coefficient magnitude, it also introduces shrinkage bias: even after the active atoms are correctly identified, the optimization objective favors smaller nonzero coefficients to reduce the penalty. Using these shrunken coefficients directly for decoder updates can underestimate the contributions of the selected atoms. We therefore retain the nonzero coefficients identified by FISTA and refit them without the l1 penalty, following post-selection refitting procedures in sparse estimation. Collecting the indices of non-zero coefficients in the support, we solve a minimized reconstruction problem constrained to the support and nonnegativity. Thus, refitting adjusts only the magnitudes of the selected coefficients without activating new atoms.
Synthetic Benchmarks: 16 Configurations of Dictionary Recovery
To study the conditions under which existing SAE methods recover the underlying dictionary and coefficients, and whether ConSDL improves recovery where they fail, we use controlled synthetic data with a known ground-truth decomposition. In the baseline setting, we use activation dimension 768, dictionary width 3072, and an expected 60 active atoms per sample. We vary the ground-truth dictionary width over four settings and vary atom-selection imbalance using Zipf exponents. This results in 16 synthetic configurations. We compare a standard ReLU SAE with TopK, BatchTopK, Matryoshka SAE, MP-SAE, and iSAE. For each synthetic configuration, we generate 10^6 observations and train each method for five epochs, repeating each experiment with three random seeds.
We evaluate dictionary recovery using decoder cosine similarity after Hungarian matching to the ground-truth dictionary. Coefficient recovery is measured by mean coefficient correlation under the same matching. Across all 16 synthetic settings, ConDL achieves near-perfect recovery of both the ground-truth coefficients and dictionary, including conditions where existing SAE methods degrade. This shows that incorporating per-sample sparse coding into dictionary learning not only addresses the coefficient-inference gap but also improves dictionary recovery when incorporated into dictionary learning. By recovering nearly the same ground-truth decomposition across runs, ConSDL also achieves high cross-seed consistency. Strong correlations exist between cross-seed consistency and ground-truth recovery, supporting cross-seed consistency as a complementary measure for real model activations where the ground truth is unavailable.
Cross-Seed Consistency Across Four Vision and Language Models
We next evaluate whether the gains observed in synthetic settings extend to foundation-model activations in vision and language domains. Since ground-truth decompositions are unavailable, we use cross-seed consistency as the primary evaluation criterion. For vision, we train SAEs on penultimate-layer activations from CLIP ViT-B/16 and DINOv2. For language, we train SAEs on layer-9 activations from GPT-2 Small and Pythia-160M. We evaluate dictionary widths of 4x, 8x, and 16x the activation dimension, each at three sparsity levels. Because ground-truth decompositions are unavailable, we evaluate cross-seed consistency of the dictionary and coefficients by training each configuration with three random seeds and computing DCS and MCC between independently trained runs.
ConSDL is more consistent than baselines across vision and language backbones. Averaged across backbones, ConSDL achieves a decoder cosine similarity of 0.84 and a mean coefficient correlation of 0.78, corresponding to relative gains of 17.6% and 35.9% over the baseline average. Compared with the strongest baseline for each metric, ConSDL improves dictionary and coefficient consistency by 1.9% and 13.3% on average. No single baseline is competitive on both metrics: MP-SAE achieves comparable decoder cosine similarity but substantially lower mean coefficient correlation, while ReLU and TopK achieve stronger coefficient consistency among the baselines but lower decoder cosine similarity. ConDL maintains high consistency in both the learned dictionaries and coefficients across vision and language backbones.
Duplicate atoms appear across several baselines, particularly ReLU and MP-SAE, while TopK-based methods and iSAE contain many dead atoms in some settings. As shown in dictionary utilization panels, ConSDL maintains high usable atom fractions across backbones, dictionary sizes, and sparsity levels. Averaged across backbones, ConSDL achieves a usable atom fraction matching the strongest baseline. Reconstruction quality remains at a similar level to the baselines across configurations.
Sparse Probing: Maintaining Interpretability Under Improved Consistency
We use sparse probing to test whether class-relevant information can be captured by a small number of dictionary atoms. Concentrating such information in a few atoms can make the learned representation easier to inspect and interpret. We evaluate vision models on six image classification datasets and language models on 113 binary tasks from SAE-Probes, using the top one or five most discriminative latents. Across vision and language models, ConSDL performs close to the strongest SAE baselines, achieving the best results in several settings and otherwise differing only marginally. This suggests that the improved cross-seed consistency of ConSDL does not come at the cost of information localized to a small number of dictionary atoms.
Limitations and Trade-offs
Coefficient refitting stabilizes dictionary learning. Comparing per-sample sparse coding with refitting, the former requires a larger l1 penalty to reach a similar sparsity level, especially on the language model. This stronger sparsity penalty is accompanied by many dead atoms and substantially lower reconstruction quality. Refitting removes the dead atoms and raises reconstruction quality on both backbones. The co-active similarity regularizer, when used alone without refitting, reduces dictionary redundancy but yields lower decoder cosine similarity than the full method. The full ConSDL method combining both components achieves the best cross-seed consistency while maintaining high dictionary utilization and reconstruction quality.
Computational cost increases with per-sample sparse coding relative to amortized encoder inference. Each input now requires an optimization loop before its coefficients contribute to the decoder update. The alternating optimization scheme, where coefficients and dictionary are held fixed at each stage, mitigates this burden but does not eliminate it. The authors note that further engineering could reduce the overhead, especially for deployment scenarios where inference latency matters.
Practical Reproducibility for Developer Workflows
For a working developer, the primary benefit of ConSDL is reduced variability in sparse decompositions across training runs. This means that analyses such as concept discovery, bias audits, or feature attribution are more likely to produce replicable results without requiring additional post-hoc alignment procedures. The code, weights, and project page are publicly available, enabling immediate experimentation. A developer can replace the learned encoder in an existing SAE training loop with per-sample sparse coding, add coefficient refitting to correct shrinkage, and apply the co-active similarity regularizer to reduce redundant atoms. The resulting dictionary and coefficient consistency should improve, particularly on configurations with larger dictionaries and long-tailed atom selection distributions.
Sparse probing performance is preserved under ConSDL, indicating that the method does not degrade the localized information that makes SAE-based inspection useful. Across vision and language models, ConSDL achieves results comparable to the strongest baselines, with marginal differences that do not follow a consistent pattern of improvement or degradation.