Test-time alignment has emerged as a compelling alternative to costly retraining, allowing a frozen language model to adapt to a new reward without updating its parameters. One family of approaches, known as pre-logit steering, adds small learned vectors to the model's hidden states just before the final classification head. These interventions can be optimized at inference time to maximize a reward signal, but there is a persistent problem: without proper regularization, steering can push the model's output distribution far from its original behavior, degrading coherence and diversity. A team at Johns Hopkins University introduces a principled solution grounded in information geometry. The paper, "Minimally Invasive Steering of Language Models," was submitted September 24, 2026.
The Problem with Unregularized Steering
Pre-logit steering works by adding a position-specific vector to each layer of the model's final hidden states before they enter the language-model head. Because the frozen model's parameters never change, the optimization only needs to search over these steering vectors, avoiding expensive backpropagation through the transformer body. This makes the approach computationally attractive and compatible with any frozen model.
The challenge is that optimizing steering vectors to maximize reward without constraints is risky. The model's hidden states occupy a low-dimensional manifold that corresponds to well-formed language. Aggressive steering can push those states off the manifold, producing outputs that score well on the reward but are gibberish, repetitive, or incoherent. Standard KL regularization helps, but computing the true sequence-level KL gradient is difficult because steering at one position affects the distribution of future prefixes, introducing a score-function term with high variance.
The authors frame the problem as reward maximization with a penalty on deviation from the reference policy. The natural measure of deviation is KL divergence, which compares the steered policy's output distribution to the original. But the exact gradient of the sequence-level KL has two components: an analytic term that captures the direct effect of steering on the current token distribution, and a score-function term that captures the indirect effect through the prefix distribution. The latter is noisy to estimate and complicates optimization.
Distributional Sensitivity Through the Fisher Information
The key insight is that the local geometry of KL divergence can be captured by the Fisher information matrix. For a given token position and prefix, the Fisher matrix describes how sensitive the token distribution is to perturbations in the hidden state. A steering direction that causes large changes in token probabilities incurs a high KL cost, while a direction that barely shifts the distribution is cheap. This is a more nuanced penalty than simply constraining the Euclidean norm of the steering vector.
The authors expand the token-level KL divergence to second order and show that the leading term is a quadratic form defined by the Fisher information matrix. Specifically, the Fisher matrix equals the transpose of the model head times a covariance matrix times the model head. This has a natural interpretation: the quadratic penalty equals half the variance of the logit perturbation induced by the steering vector, averaged over the reference token distribution. A constant shift of all logits incurs zero cost because it leaves the token probabilities unchanged — an important property that an isotropic penalty ignores.
To get fixed cost matrices that do not depend on the current prefix, the authors average the Fisher matrix over reference-policy prefixes. This gives a single cost matrix per position that captures the average distributional sensitivity. The resulting quadratic penalty is computationally efficient to evaluate and differentiate, requiring only matrix-vector products with the frozen model head.
The Gradient Decomposition and Its Implications
The paper derives an exact decomposition of the sequence-level KL gradient into two terms. The first, called the analytic Fisher term, equals the expected Fisher matrix times the steering vector at the current position. The second, the suffix score-function term, captures how steering at position t changes the future prefix distribution and thereby affects the KL contributions at later positions.
The crucial theoretical result is that for a fixed generation horizon, the suffix term is second order in the steering magnitude. Near the origin, where steering vectors are small, this term becomes negligible compared to the first-order analytic term. This means that the frozen-Fisher gradient approximates the full KL gradient with error that scales quadratically with the steering magnitude.
The authors also compare three candidate Fisher matrices: one evaluated at zero steering with reference prefixes, one evaluated at the current steering with steered prefixes, and one evaluated at zero steering with steered prefixes. They prove that all three agree with the full KL gradient to first order, and they bound the differences between them. This theoretical result justifies using the simplest and most practical choice — the frozen-reference Fisher estimated from initial rollouts — and reusing it throughout optimization without needing to re-estimate at each step.
The MISVO Algorithm
The resulting algorithm, Minimally Invasive Steering Vector Optimization or MISVO, optimizes steering vectors using stochastic gradient ascent. At each iteration, it samples multiple responses from the current steered policy, evaluates their rewards, and computes two gradient components: a reward gradient estimated via score-function methods, and a regularizer gradient computed analytically from the Fisher quadratic.
The reward gradient at each position is computed as the sum over samples of the centered reward times the gradient of the log probability of each token, projected through the model head. The regularizer gradient is simply the Fisher matrix times the steering vector. The two are combined with a regularization weight that controls the reward-deviation trade-off, and the steering vectors are updated simultaneously at all positions.
An important practical detail is that the Fisher matrix is estimated once from the initial rollouts, when the steering vectors are zero and the policy is the reference policy. This estimate is reused throughout optimization, so the computational cost of Fisher estimation does not grow with the number of iterations. The matrix can also be applied in a matrix-free manner, avoiding the need to store a full V-by-d matrix for each position, where V is the vocabulary size and d is the hidden dimension.
Experimental Results
The authors evaluate MISVO on two tasks: SHP, an open-ended preference generation benchmark using a public reward model, and MBPP+, a program synthesis task with an executable verifier. Experiments span four models ranging from 1.2 billion to 14 billion parameters — LFM2.5-1.2B, Gemma3-4B, Llama-3-8B, and Phi-4.
Across seven model-task settings, MISVO achieves the highest mean reward in six. On SHP, it outperforms both Best-of-N and AISP on three of four models. On MBPP+, MISVO leads on all three evaluated models, with the largest gain on the smallest model, LFM2.5-1.2B, where it improves the pass rate from 70.3 percent to 73.9 percent.
Crucially, the reward gains do not come at the cost of degraded generation quality. Diversity and coherence scores remain close to those of Best-of-N. On SHP, MISVO's diversity score is 0.719 and its coherence is 0.654 on LFM2.5-1.2B, compared to Best-of-N's 0.688 and 0.671 respectively — a modest trade-off that keeps generation quality intact.
A direct measurement of distributional deviation provides further evidence of the method's minimal invasiveness. When measuring token-level KL divergence under reference-policy prefixes, MISVO achieves a KL of 47.5 on LFM2.5-1.2B/SHP, compared to 303.9 for AISP and 162.2 for unregularized steering. The AISP method, despite achieving slightly higher reward on some settings, pushes the model's distribution far from the reference — the KL is roughly six times larger than MISVO's. This suggests that MISVO's Fisher-based penalty is effective at keeping the steered policy within a controlled neighborhood of the original.
The reward trajectories reveal an additional pattern. MISVO's within-step mean reward increases over the course of optimization, indicating that the steered policy is genuinely improving. In contrast, AISP's mean reward declines at later iterations, a pattern consistent with oversteering.
Limitations and Open Questions
The theoretical guarantees are local. The second-order bound on the suffix term holds near the origin, and the first-order agreement between the Fisher gradient and the full KL gradient does not guarantee accuracy for large interventions or along long optimization trajectories. The empirical deviation metric also uses reference-policy prefixes rather than the true sequence-level KL, which means it does not capture all changes in prefix visitation induced by steering.
On the SHP task, the same reward model is used for both optimization and evaluation, which limits conclusions about independent preference quality. The method also incurs iterative optimization overhead and, in the authors' implementation, higher peak memory than the baselines due to the need to store hidden states for Fisher computation.
The framework assumes a fixed generation horizon and a single prompt. Extending it to variable-length generation or to settings where the steering vectors must be adapted per prompt adds complexity. The matrix-free Fisher product approach addresses some memory concerns, but the vocabulary-sized intermediates remain a bottleneck that the authors acknowledge as a direction for future work.
Overall, MISVO represents a theoretically grounded approach to test-time steering that balances reward maximization against distributional deviation using the natural geometry of KL divergence rather than ad-hoc regularization. By deriving an exact gradient decomposition and establishing first-order agreement between the computationally convenient Fisher surrogate and the full KL gradient, the paper provides both the theoretical foundation and the practical algorithm for steering language models with surgical precision.