The paper investigates a recurring observation in language model analysis: ablating a component often prompts other components to appear to adjust and compensate. This phenomenon, labeled self-repair, has been documented across model families but its mechanism has remained unclear. The most systematic prior study concluded that self-repair is noisy and unlikely to admit a single explanation. This paper argues that a single mechanism underlies the effect: a gain present before any ablation.
The Counterweight Hypothesis
Any intervention on a causally important component can be viewed as a point on a coordinate axis λ, the signed strength of a counterfactual contrast. Conventional ablation methods are uncalibrated points on this axis. The paper introduces the affine law governing causal repair response for a fine-grained unit r: E_r(λ) = own_r + γ_rλ. The slope γ_r is a fixed coefficient that consistently influences the model, with or without ablation, and its sign determines whether the unit counteracts or reinforces the removed signal. This formulation recasts what appears as repair as the operation of pre-existing counterweights.
Affine Law for Causal Repair
The affine law E_r(λ) = own_r + γ_rλ captures the repair response as a function of the contrastive signal λ. The term own_r represents the unit's baseline contribution, while γ_rλ captures how the counterweight responds to the intervention. The coefficient γ_r is fixed across ablation and no-ablation conditions, and its sign indicates the unit's role: positive γ_r means the unit counteracts the removed signal, negative γ_r means it reinforces. This affine structure explains why repair magnitudes are predictable from model architecture alone.
Cross-Model Empirical Validation
On a factual-verdict task across four models from distinct families Gemma Qwen LLaMA and Mistral the authors identify components including MLP neurons OV neurons and singular directions that follow this affine law. 68 of 81 downstream directions in all follow the law. Moreover the magnitude of γ_r can be anticipated from the fixed weights. This extensive validation across model families demonstrates that the counterweight effect is a widespread structural property, not limited to a single architecture or training regime.
IOI Circuit Analysis: GPT-2 Small Heads
On the IOI circuit of GPT-2 Small seven of the ten heads the intervention can reach follow the law and all seven are counterweights. This clean case study demonstrates the affine law's predictive power in a minimal model where the contrastive signal emerges at the core. The fact that every intervened head that follows the law operates as a counterweight reinforces the paper's central claim: what may appear as self-repair is counterweight performing its usual operation.
Anticipating γ_r from Fixed Weights
The paper shows that the magnitude of γ_r can be anticipated from the fixed weights of the model. This means that before any ablation experiment the counterweight strength is already encoded in the parameters. The ability to predict repair magnitude from untouched weights opens the door to pre-experiment analysis of component importance and reduces the interpretive burden of ablation studies.
Reframing Ablation Interpretation
The framework reframes how researchers interpret ablation studies. Instead of searching for repair mechanisms the pre-existing counterweight structure provides a calibration baseline. The λ axis offers a coordinate system for locating any intervention, and the affine law enables magnitude prediction without additional experimentation. This shift in perspective—from post-hoc repair explanation to pre-existing counterweight operation—simplifies the interpretive pipeline for ablation analysis across language model families.
LoopCD demonstrates that recurrence itself provides the weak-and-strong prediction pairing needed for contrastive decoding, eliminating the need for separate reward models or value networks. The approach is training-free and can be applied at inference time with minimal disruption to existing pipelines.
Read the paper on arXiv