Catastrophic forgetting remains one of the fundamental obstacles to continually updating large language models. Standard fine-tuning overwrites the weights that encode prior capabilities, and rehearsal-based methods require storing or regenerating old data—often impractical at scale. Chatterjee and Makras reframe forgetting as a geometric problem in the input space of each weight matrix. They show that gradient-based updates are suboptimal under a natural retention objective, and they propose Local Support Learning (LSL), a post-training framework that pairs a standard weight adapter with a Gaussian Mixture Model gate. The gate activates the adapter only on activations from the current training distribution, keeping updates local to that distribution while staying closed on data from prior phases. Experiments on LLMs up to 7 billion parameters demonstrate that LSL retains both pretrained and fine-tuned capabilities across multiple sequential training phases, with low memory and compute overhead, robustness to hyperparameters, and promising scaling behaviour.
Forgetting as a Geometry Problem
When a pre-trained model is fine-tuned on a new task, the weight matrices receive gradient updates that minimise the new loss. Viewed in the input space of a weight matrix W ∈ ℝ^{d_out × d_in}, each input activation x ∈ ℝ^{d_in} maps to an output Wx. The pre-trained weights define a linear map that has been shaped by the pre-training distribution. A gradient step on new data moves W in a direction that reduces loss on the new distribution, but this same step perturbs the map for all inputs—including those from the pre-training distribution that the model has never seen during fine-tuning.
The authors observe that the ideal retention objective is to constrain the change in Wx for inputs x drawn from prior distributions, while allowing free change for inputs from the current distribution. Gradient descent does not distinguish between these input populations; it applies the same update everywhere. This geometric perspective reveals that forgetting is not merely a side effect of optimisation but a consequence of applying global updates to a problem that is inherently local in activation space.
Local Support Learning: Adapter Plus Gate
LSL introduces two components per weight matrix during a new learning phase. First, a low-rank weight adapter ΔW (typically implemented via LoRA) that is trained normally to minimise the task loss. Second, a gating function g: ℝ^{d_in} → [0, 1] that modulates the adapter's contribution: the effective weight becomes W + g(x)ΔW. When g(x) ≈ 1, the adapter is fully active; when g(x) ≈ 0, the adapter is suppressed and the original weights prevail.
The gate must satisfy a critical requirement: it must route activations from all learning phases (pre-training, prior fine-tuning stages, current fine-tuning) while being trained only on data from the current phase. This seems paradoxical—how can a gate learn to stay closed on distributions it has never seen? The authors resolve this by modelling the gate with a Gaussian Mixture Model (GMM) fitted to the current phase's input activations. A GMM's likelihood decays rapidly away from its training data. Thus, activations from the current distribution receive high likelihood (gate open), while activations from prior distributions—lying outside the GMM's high-density regions—receive low likelihood (gate closed). The gate naturally generalises to reject out-of-distribution inputs without ever seeing them during training.
Concretely, for each weight matrix, the authors fit a GMM with K components to a buffer of input activations collected during the current training phase. The gate value for an input x is the GMM likelihood p(x) normalised by a temperature parameter. During inference, every forward pass evaluates the GMM on the incoming activations and scales the adapter accordingly. The GMM parameters are frozen after the phase ends, so the gate retains its selectivity for all subsequent phases.
Training and Inference Mechanics
During a new learning phase, the model processes batches from the new dataset. For each layer, input activations are cached to update the GMM (using online EM or a fixed buffer with periodic refitting). The adapter ΔW receives gradients through the gated output: ∂L/∂ΔW = g(x) · ∂L/∂(Wx). The gate itself is not differentiated through; it acts as a fixed mask determined by the GMM. This design avoids the instability of jointly optimising the gate and adapter, and it ensures the gate's behaviour is determined solely by the geometric structure of the current phase's activations.
At the end of the phase, the GMM is finalised and frozen. The adapter weights are merged into the base model (or kept separate for future phases). When the next phase begins, a new adapter and a new GMM are initialised for each layer. The previous GMMs remain in place, each guarding its own adapter's domain. An input activation passes through a cascade of gates: the current phase's gate (likely open), the previous phase's gate (likely closed), and so on. The effective weight update at inference time is the sum of gated adapters: W + Σ_i g_i(x)ΔW_i.
Memory overhead is modest: each GMM stores K mean vectors (d_in each), K covariance matrices (diagonal or full), and K weights. For d_in = 4096 and K = 16, a diagonal-covariance GMM requires roughly 130 KB per layer—negligible compared to the model weights. Compute overhead is a single GMM likelihood evaluation per layer per token, which is a small fraction of the matrix multiply cost.
Experimental Results: Retention Across Phases
The authors evaluate LSL on language models up to 7B parameters across multiple sequential fine-tuning phases. The benchmark spans diverse tasks: scientific question answering, mathematical reasoning, coding, and general instruction following. Each phase fine-tunes on a new dataset; retention is measured by evaluating on all previous tasks after each phase.
Key results: LSL reduces forgetting to near-zero on most benchmarks. On a 7B model undergoing five sequential fine-tuning phases, the average performance drop on prior tasks is under 2% relative to the single-task fine-tuned baseline, compared to 15–30% drops for standard LoRA and 5–10% for experience replay with a 1% memory budget. LSL matches or exceeds replay-based methods without storing any prior data. The GMM gate's temperature parameter controls the specificity–generality trade-off; a single default value works well across tasks and model sizes, confirming robustness.
Ablations confirm the geometric intuition. Replacing the GMM with a simple distance-to-centroid gate degrades performance, as the GMM's elliptical contours better capture the anisotropy of activation distributions. Training the gate jointly with the adapter (rather than freezing it) leads to instability and worse retention. Using a single global gate across all layers (instead of per-layer GMMs) fails because different layers' activation spaces have different geometries.
Limitations and Trade-offs
The approach assumes that the activation distribution of each phase is sufficiently concentrated for a GMM to model it effectively. For tasks with extremely diverse or multi-modal input distributions, a fixed K may underfit, requiring more components or a richer density estimator. The per-layer GMMs add a small but non-zero inference latency; for latency-critical applications, the gate evaluation could be fused with the matrix multiply or approximated. The method does not address forgetting in the embedding layers or output head, where activation geometry differs; the authors apply standard LoRA there with a separate strategy. Finally, the theoretical analysis of the retention objective is preliminary—while the geometric framing is compelling, a formal regret bound or convergence guarantee for the gated updates remains open.
What This Means for Practitioners
For teams deploying LLMs that need periodic updates—new domains, updated knowledge, corrected behaviours—LSL offers a practical path to continual learning without data replay. The per-layer GMM gates are simple to implement, add minimal overhead, and can be dropped into existing LoRA pipelines. The hyperparameter robustness (single temperature, default K) reduces tuning burden. The method scales to 7B parameters in the paper; the geometric principles suggest it should scale further, as activation concentrations tend to sharpen in larger models. Open questions include optimal GMM refitting schedules, handling distribution shift within a phase, and combining LSL with parameter-efficient methods beyond LoRA (e.g., prefix tuning). The code and checkpoints are released, enabling immediate experimentation.
The deeper contribution is the shift in perspective: forgetting is not an inevitable consequence of gradient descent but a solvable geometric mismatch. By making updates local to their supporting data, LSL turns the activation space into a partitioned map where each adapter owns its territory. This principle—local support, global composition—may extend beyond continual learning to multi-task fusion, model merging, and efficient ensemble inference.