When an LLM memorizes and regurgitates copyrighted text, personal information, or other undesirable training content, the clean solution is retraining from scratch without that data. But retraining a large language model is prohibitively expensive. Machine unlearning offers a shortcut: modify the trained model's parameters to suppress the unwanted knowledge while preserving everything else. The problem is that most existing unlearning methods take a blunt approach, updating the entire model or a fixed subset of parameters. This creates two failures. First, the updates diffuse across the model, degrading utility on tasks that have nothing to do with the forgotten content. Second, when the model is later quantized for deployment, small diffuse updates get erased by low-bit rounding, and the forgotten knowledge re-emerges. FOM-UL from Florida International University and Oak Ridge National Laboratory proposes a targeted alternative: compute a significance score for each transformer layer, select only the layers that matter most for the forget set while being least sensitive to the retain set, and update only those.
Why global updates fail under quantization
The core issue is mathematical. When you unlearn by adding small perturbations across all parameters, each individual change is tiny. Post-training quantization maps full-precision weights to low-bit representations by rounding to the nearest bin. If a parameter's unlearning update is small enough that it stays within the same quantization bin as the original value, the quantized model functionally reverts to the pre-unlearning state. The paper formalizes this: an update to coordinate j is erased by quantization when the rounded values of the original and unlearned parameters are identical. When this holds for a large fraction of edited coordinates, the quantized unlearned model becomes functionally closer to the quantized original model, allowing suppressed knowledge to re-emerge.
FOM-UL addresses this by concentrating updates into a small number of layers rather than spreading them across the full model. Larger, more concentrated parameter shifts are less likely to fall within a single quantization bin, making them more resilient to the rounding operation. The paper calls this an empirical path toward quantization-resilient unlearning, and the results support the claim.
The forget-to-retain significance score
The selection criterion is elegant. For each transformer layer, FOM-UL computes two gradient magnitudes: the L2 norm of the gradient of the forget loss with respect to that layer's parameters, and the L2 norm of the gradient of the retain loss. The significance score is their ratio: Sig(l) = I(l) / (Ir(l) + epsilon), where I(l) is the forget gradient magnitude and Ir(l) is the retain gradient magnitude. Layers with high scores are those that strongly influence the forget set while being comparatively insensitive to the retain set. These are the layers where updates will be most effective at erasing targeted knowledge with minimal collateral damage.
The initial selection picks all layers above a threshold tau. If the forgetting criteria are not met (e.g., verbatim memorization remains above 0.05), the set expands by adding the next most significant layer. This iterative expansion prevents aggressive updates early on and stops when forgetting converges or a maximum epoch count is reached.
The paper also computes a layer-wise ablation score: for each layer, it zeros out that layer's contribution to the residual stream and recomputes the next-token probability. This identifies which layers most influence the model's predictions on specific inputs. Combined with the gradient-based significance score, this dual analysis ensures that selected layers are genuinely responsible for encoding the targeted knowledge, not just statistically correlated with it.
Three loss components for targeted forgetting
Updates are guided by three losses applied only to selected layers. The forgetting loss increases the model's loss on forget-set samples, pushing it away from reproducing targeted content. The mismatch loss diverges the model's outputs from its original predictions on forget samples, ensuring the model doesn't just become uncertain but actively produces different outputs. The retain loss decreases the loss on retain-set samples, preserving utility on knowledge that must be kept. Layers not in the selected set remain completely frozen.
The ablation study confirms that all three components are necessary. Forget-only achieves low memorization (M1 = 1.56) but high privacy leakage (M3 = 8.31). Retain-only preserves utility (M4 = 4.62) but achieves weak forgetting (M1 = 2.28). Mismatch-only improves utility but introduces privacy instability (M3 = -6.44). The full FOM-UL combination achieves the best balance: M1 = 1.24, M3 = 2.04, M4 = 2.90.
Results across three benchmarks and three model families
On TOFU-World Facts, FOM-UL combined with NPO achieves the lowest verbatim memorization (M1 = 1.58-1.68 across GPT-2, Llama-3.2-1B, and Gemma-3-1B) while maintaining utility (M4 = 2.28-2.42), compared to the vanilla model's M4 = 2.90. The privacy leakage score (M3) drops to 2.66-2.84, far below the vanilla model's 6.94-8.10 and competitive with or better than SURE+NPO (M3 = 3.40-3.64).
On NEWS, KnowUnDo, and BOOKS datasets using Llama-3.2-1B, FOM-UL consistently matches or ties the best memorization scores while substantially reducing privacy leakage. On NEWS: M1 = 1.56, M3 = 2.64, M4 = 2.90. On KnowUnDo: M1 = 1.58, M3 = 2.64, M4 = 2.90. On BOOKS: M1 = 1.56, M3 = 2.64, M4 = 2.90. The vanilla model scores M1 = 5.80, M3 = 8.80, M4 = 2.90 across all three datasets, so FOM-UL reduces memorization by roughly 73% while preserving utility at the original level.
The comparison with LUNAR is particularly instructive. LUNAR achieves strong memorization suppression (M1 = 1.22-1.56) but at the cost of lower utility (M4 = 1.86-2.14). FOM-UL achieves comparable memorization (M1 = 1.24-1.68) with significantly higher utility (M4 = 2.28-2.90). The targeted layer selection allows FOM-UL to suppress memorization without the utility degradation that global or fixed-selection methods incur.
Quantization resilience
Under 8-bit post-training quantization, FOM-UL maintains M1 = 2.04, compared to GA+GDR at 5.32, NPO+GDR at 3.12, and LUNAR at 3.14. Under 4-bit quantization, FOM-UL achieves M1 = 2.44, compared to GA+GDR at 5.94, NPO+GDR at 3.68, and LUNAR at 3.88. The gap between FOM-UL and baselines widens under more aggressive quantization, confirming that concentrated layer updates are more resistant to bin-rounding erasure than diffuse parameter changes.
The paper notes that FOM-UL's M3 values under quantization are negative, indicating deviation from the retrained privacy baseline. The authors interpret this as evidence of privacy-behavior shift under aggressive quantization rather than improved privacy, and point to the memorization and utility metrics as the primary robustness evidence.
Adversarial robustness
Under jailbreak-style adversarial prompts designed to extract forgotten content, FOM-UL achieves the lowest attack leakage rate at 11.6%, compared to 16.5% for SURE+NPO and 19.8% for LUNAR. This indicates that the concentrated updates are harder to reverse through prompt engineering than diffuse updates, likely because the critical layers have been more thoroughly modified.
Runtime and efficiency
FOM-UL updates only 7 million parameters out of a 7 billion parameter model (Llama-2), requiring 8 GB of GPU memory and 20 minutes of runtime. Compare this to full-model GA+GDR: 7 billion trainable parameters, 16 GB memory, 4 hours. Or KLD: 7 billion parameters, 32 GB memory, 4 hours. FOM-UL is competitive with parameter-efficient baselines like SURE+NPO (1.7M params, 8 GB, 15 min) and LUNAR (1.75M params, 8 GB, 20 min) while achieving stronger forgetting-utility trade-offs.
The efficiency gain comes directly from the layer selection. By freezing most of the model, FOM-UL reduces the gradient computation, optimizer state, and memory footprint to a fraction of full-model methods. The iterative expansion means it starts with even fewer parameters and adds only as needed.
What this means for deployment
The practical scenario is straightforward. A company deploys a quantized LLM and receives a GDPR takedown request for specific training data. They need to unlearn that data without retraining, and the unlearned model must remain quantized for inference efficiency. Existing methods either fail to forget completely (residual memorization remains high), degrade utility on unrelated tasks, or see the forgetting undone by quantization. FOM-UL addresses all three failure modes by concentrating updates where they matter and making those updates large enough to survive bin-rounding.
The framework is compatible with any standard unlearning loss (GA, NPO, KLD), so it can be plugged into existing pipelines. The layer selection is computed once per forget set and can be cached. The iterative expansion adds minimal overhead. For organizations processing frequent unlearning requests, this is a practical path to compliant deployment without the cost of full retraining.
The authors are careful not to claim formal guarantees of erasure. FOM-UL is an approximate unlearning method that reduces memorization empirically. But the combination of stronger forgetting, better utility preservation, quantization resilience, and adversarial robustness makes it a meaningful advance over the current state of the art, particularly for real-world deployment where quantization is the norm rather than the exception.