Low-Rank Adaptation (LoRA) specializes pretrained models by learning compact updates while keeping the original weights frozen. However, freezing pretrained parameters does not preserve pretrained behavior: the added update can improve the target task while compromising capabilities outside the adaptation distribution. This creates a practical challenge for deployment, where specialization and capability retention both matter. An important question is whether a completed adapter can better satisfy these objectives through changes to how its learned directions are used.
We examine this question by separating an adapter's directions from their gains, building on a broader line of parameter-efficient methods that exploit weight decomposition or directional structure. The decomposition B A = U diag(σ) V⊤ expresses the update through learned singular directions and the strengths assigned to them. Although LoRA constrains the update rank, it does not require adaptation strength to be distributed evenly across these directions. Our measurements reveal substantial concentration in trained adapters, with directional output energy dominated by a small subset of the available components. We describe this uneven allocation as adaptation imbalance and investigate its deployment consequences by changing gains while keeping the learned directions fixed.
Our central argument is that learning where to adapt does not ensure that adaptation gains are well balanced. Training jointly determines directions and gains through a source-task objective, but the resulting configuration can remain improvable for specialization and broader capability retention. Prior work establishes the feasibility of post-training refinement through weight interpolation, spectral pruning, and calibration-gradient-guided reweighting. We investigate a simpler intervention: a fixed normalization rule that revisits the gains of a completed adapter using only its parameters.
LoRA-Norm: Post-hoc spectral rebalancing
We propose LoRA-Norm, a post-training normalization method that retains learned directions while rebalancing their gains. LoRA-Norm combines spectral rebalancing, a fixed nonlinear transformation of singular values, with nuclear-norm restoration, which preserves the original total spectral mass. It requires no calibration data or additional training and introduces no inference overhead.
For a frozen weight matrix W₀ ∈ ℝ^{d_out × d_in}, LoRA parameterizes the adapted weight as W = W₀ + c B A, where B ∈ ℝ^{d_out × r}, A ∈ ℝ^{r × d_in}, and c is the original adapter scaling. Following the fixed-direction editing formulation, we decompose the trained update: D = B A = U diag(σ) V⊤. Only the nonnegative gains change from σ to t, keeping their pairing with the source columns of U, V fixed, even when the transformation changes their relative ordering. The base weights, adapter scaling, and stored rank budget remain unchanged.
The normalization procedure consists of two stages. First, we normalize the source spectrum by its Euclidean norm and apply elementwise quintic updates in two stages (a fast stage that strongly amplifies small positive inputs, and a stable polynomial that has f_stable(1) = 1 and f_stable'(1) = 0, refining gains near unity). After clipping numerical negative values, we restore the original nuclear budget and reconstruct balanced low-rank factors, ensuring ‖c B' A'‖_* = ‖c B A‖_* in exact arithmetic. This is a fixed parameter-space budget, not preservation of the Frobenius norm or functional update energy.
LoRA-Norm therefore requires no calibration data or additional training and introduces no inference overhead. The edited update is reconstructed as standard low-rank factors with the original adapter configuration and scaling.
Experimental evaluation
We evaluate 18 trained adapters spanning two backbones (Qwen3-8B and Llama-3.1-8B-Instruct) and three adaptation tasks (Magicoder, MetaMath, and Tulu). LoRA-Norm improves average target-task performance and capability retention, exceeding the evaluated spectral-pruning and gradient-guided editing configurations on both measures.
Table 1 shows the primary results. Averaged over 18 checkpoints, LoRA-Norm raises Target from 68.00 to 72.08 (+4.09 percentage points) and Off from 69.55 to 72.79 (+3.24), while reducing FG from 2.71 to 0.34. Target improves on 16 checkpoints, ties on one, and decreases on one; Off improves on 15 and decreases on three; FG improves on 14 and ties on four.
Compared with post-hoc editing methods, LoRA-Norm has the highest mean Target and Off and the lowest FG among the edited adapters. It exceeds PARA (with ε = 0.90) by 3.31 Target and 2.78 Off points; PARA additionally compresses the adapter, retaining 38.16–80.16% of parameters across its settings. The evaluated Spectral Surgery configuration improves Target over LoRA by 0.77 points but reduces Off by 1.27. LoRA-Norm exceeds it by 3.32 Target and 4.51 Off points without calibration data.
Sensitivity to HNS iteration counts examined on six source checkpoints, one per backbone for each of the three primary adaptation tasks, varying K_f ∈ {2, 4, 8} and K_s ∈ {0, 1, 2}. Averaged over these six checkpoints, all nine nonzero configurations yield gains of +3.42 to +3.62 percentage points over LoRA. The main setting 4+1 yields +3.57 points, within 0.05 points of the largest observed mean. Increasing either the number of fast or stable steps does not monotonically improve performance.
Analysis of gain redistribution
Our analysis also clarifies what this improvement means. Preserving nuclear norm does not preserve update magnitude, and a direction's functional contribution depends on both its gain and the inputs that activate it. Controls matched to LoRA-Norm's activation-weighted update energy recover much of its benefit, while stronger functional equalization provides no consistent additional gains.
We construct three controls from each module's actual LoRA-Norm spectrum: Scalar-F matches its Frobenius norm, Scalar-E matches its functional energy, and Flat-E uses equal parameter gains at the same functional energy. Frobenius matching leaves a performance gap (LoRA-Norm exceeds Scalar-F by 1.44 Target points and 1.23 Off points, while reducing FG by 0.77 points). Energy matching reproduces much of the benefit: the LoRA-Norm-minus-Scalar-E differences are −0.62 Target points and +0.32 Off points, with neither interval excluding zero. Exact parameter flattening (Flat-Nuclear) is also competitive, improving Target on 16 of 18 checkpoints and trailing LoRA-Norm by only 0.15 points on average, with paired wins/ties/losses of 8/4/6 against LoRA-Norm.
These results show that the same learned directions can support better deployment performance after gain normalization, without any additional training. They also clarify that balancing adapter gains and equalizing their responses are distinct objectives. Directional concentration is useful for characterizing an adapter, but uniformity alone is insufficient to judge whether an edit improves specialization and retention.
Conclusion
In a nutshell, our contributions are summarized as follows:
- Empirical insight. We characterize adaptation imbalance in trained LoRA adapters and show that their learned directions can support improved specialization and capability retention through post-training gain adjustment.
- Practical method. We introduce LoRA-Norm, a fixed spectral normalization procedure that preserves learned directions and per-module nuclear mass, requiring neither calibration data nor additional training or inference operations.
- Functional analysis. We examine gain redistribution alongside update strength through matched-strength controls and activation-aware interventions, showing why stronger response equalization is not a reliable objective for adapter refinement.
LoRA-Norm improves average target-task performance and capability retention, revealing that the same learned directions can support better deployment performance after gain normalization, without any additional training. Stronger functional equalization brings no consistent additional gains, revealing that balancing adapter gains and equalizing their responses are distinct objectives.