Every agent that operates with real autonomy carries a silent liability: the possibility of catastrophic failure, however improbable. A coding agent might accidentally delete a production database. A scientific agent might hallucinate a chemical formula that, if acted upon, could cause harm. These events may occur only once in a billion trajectories, but as agents are deployed at scale, even astronomically rare outcomes become statistically inevitable. Safe deployment therefore depends not on whether these events can occur, but on how often they might. Quantifying that frequency is the problem that a new paper addresses, and it does so with a method that achieves over 800 times the compute-weighted efficiency of brute-force sampling.
The Rare Event Problem in Agent Systems
An LLM-based agent is a stochastic system. At each step, it samples tokens from a probability distribution, and the resulting trajectory is one draw from a vast combinatorial space. Most trajectories are unremarkable: the agent completes its task correctly, or fails in uninteresting ways. But somewhere in the tail of the distribution lie trajectories where the agent does something catastrophic: it follows a malicious prompt injection, produces unsafe code, or takes an irreversible action that causes real-world harm.
Estimating the probability of these tail events is fundamentally difficult. Naive Monte Carlo sampling, which generates trajectories and counts how many times the event occurs, is computationally prohibitive. If an event has a probability of 10^-9, you need on the order of a billion samples to observe even a single occurrence. At typical inference costs of a few cents per trajectory, that is an infeasible budget.
The standard solution in statistics is importance sampling. Instead of sampling from the original model distribution, you sample from a modified distribution that makes the rare event more likely, then reweight the observations to estimate the true probability. The challenge is constructing that modified distribution. For a language model, the probability of a trajectory factorizes into a chain of conditional distributions, and shifting each one to favor the rare event requires coordinated changes across the entire sequence. A naive approach, such as simply encouraging more unusual token choices at each step, may produce trajectories that look rare but are structurally nothing like the catastrophic event you are trying to measure.
The Core Idea: Perturb the Model, Not the Input
The paper introduces a fundamentally different approach. Rather than modifying the sampling process at inference time or engineering a complex proposal distribution, the method perturbs the model's own weights to construct the proposal. The idea is that if you want to make rare events more likely, you can directly search over the space of model weights for a version of the model that assigns higher probability to those events.
This weight-space search is made tractable by representing the proposed model as a differentiably parameterized language model. The proposal model is not a separate architecture trained from scratch; it is a modification of the original model, parameterized in a way that allows gradients to flow through the probability of the rare event with respect to the perturbation parameters. Gradient-based optimization can then iteratively adjust the weights to amplify the target event.
The authors call their method iterative unalignment, and the term captures the essence of the approach. Standard training aligns the model to produce safe, typical outputs. Rare event estimation requires the opposite: deliberately unaligning the model so that it explores the tails of its own output distribution. Each iteration adjusts the model slightly toward a configuration that assigns higher probability to the rare event, and the gradient signal tells it how to adjust.
The Optimization Objective
The objective function combines two components. The first is a differentiable surrogate for event amplification: it measures how likely the perturbed model makes the target rare event, and gradients push the perturbation in the direction that increases that likelihood. The second is an adaptive regularization scheme that dynamically balances amplification against estimator stability.
This regularization is critical because unconstrained weight perturbation can produce a model that makes the target event likely but in a way that is uninformative about the original model's behavior. The regularizer prevents the perturbed model from drifting too far from the original, ensuring that the importance weights remain valid and the estimator remains unbiased. The adaptive scheme adjusts the strength of regularization over the course of optimization, starting with tighter constraints to maintain stability and gradually relaxing them as the search converges.
The resulting optimization produces a proposal model that is both more likely to generate the rare event and still recognizably related to the original model, so that the importance sampling correction is meaningful.
Scale of the Evaluation
The paper evaluates across a substantial experimental setup. Two model scales, approximately 120 million parameters and approximately 2.6 billion parameters, are tested. Three event families, spanning distinct types of rare behavior, are studied across more than 300 individual rare events. Some of these events have probabilities as low as 10^-9, which places them firmly in the regime where naive Monte Carlo is completely impractical.
The reference probabilities are computed with less than 10 percent relative standard error, providing a reliable benchmark against which to measure the importance sampling estimator's accuracy. This level of precision is necessary because importance sampling estimators can have high variance, and without a reliable ground truth, it is impossible to distinguish a good estimator from a lucky one.
The Efficiency Numbers
The headline result is that the IS estimator achieves over 800 times compute-weighted efficiency gains over naive Monte Carlo for events with probabilities below 10^-7. This is not a modest improvement; it is the difference between requiring billions of samples and requiring only a few million. In practical terms, a risk assessment that would take months of computation can be completed in minutes.
The efficiency gain is particularly striking because it holds across model scales and event families. The method is not optimized for a single type of rare event or a single model architecture. The gradient-based weight perturbation approach generalizes: the same optimization procedure works whether the rare event is a security failure, a factual hallucination, or a logical error, and whether the model is small or large.
The 120 million and 2.6 billion parameter models also provide a scaling perspective. If the method works similarly well at both scales, it suggests that the weight-space search does not depend on model size in a way that would undermine its applicability to larger systems.
How This Differs from Prior Work
Rare event estimation for language models has attracted growing attention. Prior approaches have included importance sampling in input space, activation-space methods that navigate the continuous representations inside the model, and exponential reweighting techniques borrowed from statistical mechanics.
The paper by Dorman et al. (2026), which appeared as an ICML 2026 oral spotlight, established a framework for systematic rare event analysis in LLMs using techniques from statistical physics, including exponentially reweighted distributions and trajectory-based observables. Parulekar and Liu (2026) explored activation-space methods that use gradient-based MCMC to navigate the internal representations of the model.
What distinguishes the RRSI-adjacent work from this paper is the level at which the perturbation operates. Input-space methods search for prompts that trigger rare behavior. Activation-space methods navigate the model's internal representations after the fact. The method in this paper operates at the weight level, searching for model parameters that make rare events likely by design. This is a more direct and potentially more efficient approach because it modifies the model itself rather than trying to find specific inputs or manipulate internal states.
The related Adaptive Layerwise Perturbation work (2026) injects perturbations into hidden states during training to stabilize off-policy RL, but it is not designed for rare event probability estimation. The rare event estimation methods from statistical mechanics, such as twisted sequential Monte Carlo, operate at the trajectory level and require complex resampling schemes.
Limitations and Honest Inferences
The method has several limitations worth acknowledging. First, the weight perturbation approach assumes that the model can be meaningfully modified in a differentiable way. For some architectures or training paradigms, the parameter space may not be amenable to the kind of gradient-based search that this method requires.
Second, the adaptive regularization scheme introduces its own hyperparameters. The balance between amplification and stability is crucial, and getting it wrong can produce either an estimator with high variance or a proposal that is too close to the original model to provide any efficiency gain. The paper does not fully explore how sensitive the results are to the regularization parameters.
Third, the evaluation is limited to three event families. While these span security, factual accuracy, and logical reasoning, there are many other types of rare events that agents can encounter: social manipulation, data exfiltration, regulatory violations. The method's applicability to these event types is not established.
Fourth, the method estimates probabilities but does not explain why the rare events occur. Knowing that an agent has a 10^-7 probability of taking a catastrophic action is useful for risk management, but understanding the mechanism behind that probability is necessary for mitigation. The paper does not address the interpretability dimension of rare event analysis.
Fifth, the weight-perturbed model may behave differently from the original model in ways that are not captured by the importance weights. If the perturbation changes the model's behavior so fundamentally that the importance correction becomes unreliable, the estimator's accuracy degrades. The regularization helps, but the paper does not provide a theoretical bound on when the estimator remains valid.
What This Means for Practitioners
For organizations deploying autonomous agents at scale, the practical implication is that risk assessment is now computationally feasible. A company that wants to understand how often its coding agent might produce unsafe code, or how often its customer-service agent might leak personal information, can now estimate those probabilities without waiting for a real-world incident to reveal the risk.
The 800x efficiency gain transforms rare event estimation from a theoretical exercise into an operational tool. Teams can run these assessments as part of their deployment pipeline, regularly checking whether the agent's rare-event profile has shifted as the model or its environment changes.
The method also has implications for safety certification. Regulatory frameworks for AI are increasingly requiring that developers demonstrate knowledge of their systems' failure modes. Being able to estimate the probability of catastrophic failures, rather than merely observing them, is a meaningful step toward evidence-based safety assurance.
The Bigger Picture
This paper sits at the intersection of several important research threads in AI safety and statistical computing. The core challenge is one that has been studied for decades in physics, chemistry, and operations research: how to estimate the probability of events that are too rare to observe directly. The innovation is adapting these techniques to the specific structure of language models, where the combinatorial explosion of trajectories makes naive sampling impossible and where importance sampling requires navigating a high-dimensional probability space.
The concept of iterative unalignment is philosophically interesting. In safety research, alignment refers to making models behave as intended. This paper argues that understanding failure requires the opposite impulse: deliberately moving the model away from its aligned distribution to explore the space of possible failures. It is a reminder that safety is not just about building better models, but about understanding the full distribution of what existing models can do.
As agents become more autonomous and are deployed in higher-stakes environments, the ability to quantify risk rather than merely react to incidents will become a core capability. This paper provides a concrete method for doing so, and the 800x efficiency gain suggests that the method is not just theoretically elegant but practically deployable.