Why Value Gating Works in Attention Has Been a Blind Men and the Elephant Problem
Gating the value pathway of attention has become a popular trick in language model pretraining. A series of recent papers report that adding a learned gate to the values improves validation loss, and they agree on the result. They disagree, sharply, on why. One study attributes the gain to added nonlinearity and sparsity. Another frames it as the mitigation of extreme-token phenomena, meaning attention sinks and the massive activations that accompany them. A third distinguishes sinks that let a head perform no operation from sinks that broadcast information, and claims gating addresses the first kind. A fourth observes that attention sinks disappear when a gate is present. None of these studies isolate one mechanism from the others, because a single gating mechanism can supply multiple capabilities simultaneously.
The paper by Richard Zhe Wang decomposes the mystery. It argues, and provides experimental evidence across matched models from 10M to 350M parameters, that value gating supplies two distinct primitives that standard softmax attention lacks. The first is abstention: the ability of an attention head to output nothing, bypassing the requirement that attention weights sum to one. The second is noise filtering: the ability of the value pathway to suppress interference from superposed features in the residual stream. These are different operations, serving different needs, and they exhibit opposite scaling behavior.
What Softmax Attention Cannot Do
Standard causal attention computes routing weights by applying softmax to query-key dot products. The softmax guarantees that weights sum to one. When a query finds nothing worth reading in its context, it must still distribute its full attention mass somewhere. Models compensate by designating a position, usually the first token, as a parking spot for unwanted mass. This is the attention sink phenomenon: a learned no-operation that the model constructs implicitly rather than being given explicitly. The larger the model, the more precisely it builds this sink, driving the first token's value toward zero and concentrating unwanted mass there.
The second gap is that the value pathway is linear end to end. In a model where features are superposed in a shared basis, meaning more features exist than there are dimensions to represent them, a read at a position that carries none of the features a head is looking for consists of a small amount of interference from every feature that is present. An MLP neuron removes such interference with its thresholding nonlinearity, but the value pathway has no comparable mechanism. A noisy read is aggregated with the same fidelity as a clean one.
Wang supplies each gap with its own direct mechanism. Abstention is provided through a learned per-head sink logit: a phantom key with a learnable logit and a value fixed at exactly zero. The phantom competes inside the same softmax as real keys, so its share of attention mass is set by the query's logits and can be large for one query and negligible for the next. This makes abstention exact and query-conditional. Noise filtering is provided through a gate on each value read, in two forms: a threshold on the read's norm, and a learned linear projection of its content.
The Scale-Dependent Crossover
The central finding is that the two primitives exhibit opposite scaling behavior. The benefit of abstention, measured as the reduction in validation loss relative to a matched baseline, declines as models grow. The benefit of noise filtering increases with scale. The crossover happens between 50M and 124M parameters.
At 10M parameters, the sink logit (abstention alone) improves validation loss by 0.0185 nats over the baseline. The norm gate (filtering alone) improves it by only 0.0037 nats, and most of that gain comes from the partial zero option the gate carries, not from filtering. When both are combined, the improvement is 0.0177 nats, essentially the same as the sink logit alone. At this scale, abstention accounts for nearly all of the gain from gating.
At 350M parameters, the situation reverses. The sink logit improves loss by only 0.0013 nats, which is within the seed spread observed at 124M. The combined arm with the projection gate improves loss by 0.0128 nats, with the filtering component contributing 0.0114 nats, five to nine times as much as the zero option. The filter now carries the weight.
The explanation is visible in the trained models. A baseline without a zero option imitates one by forming an attention sink, and larger baselines imitate it more effectively. At 124M, the baseline places 6.0% of all attention mass on the first key position, whose value norm is 0.42 times that of an ordinary read. With a sink logit, mass on position 0 drops to 0.5% and the norm ratio recovers to 0.99. At 350M, the baseline parks 11% of its mass on position 0 with a norm ratio of 0.36, while the sink-logit model places 0.7% there and routes 53% of its mass to the phantom. The demand for abstention grows with scale, but the loss benefit of supplying it shrinks because the baseline's own imitation gets closer to the real thing. Meanwhile, the filtering increment, measured as the norm gate's within-row spread (how much it discriminates among reads), grows on the same trajectory as its loss benefit.
The Two Benefits Are Additive
The second finding directly tests whether abstention and filtering are separate mechanisms. If they were two descriptions of one mechanism, supplying both would give no more than the better one. If they were fully independent, their benefits would add exactly. The reality is close to the second case, with a small overlap predicted by theory.
A value gate used alone carries a partial zero option: it attenuates each read by a scalar, and if every read is fully attenuated, the output reaches zero. This partial zero option overlaps with what the sink logit provides. Lemma 3.1 in the paper makes this precise. A value gate is exactly equivalent to a routing gate (multiplying each routing weight and renormalizing) followed by scaling the head output by a factor Z_i that is at most one. The renormalized routing conserves attention mass; the factor Z_i dissipates it. Comparing a gate in the value slot with the same gate signal in the routing slot isolates that partial zero option.
The two-by-two decomposition confirms the prediction. The overlap, defined as the sum of the two solo improvements minus the improvement from both together, is small across every scale: between 0.0024 and 0.0076 nats for the norm gate, and between 0.0029 and 0.0076 nats for the projection gate. At 10M, the overlap is at least the norm gate's entire solo benefit and two-thirds of the projection gate's. At 124M, it is under half of either. The filtering column, which measures the improvement of the combined arm over the sink logit alone, is positive at every tier from 50M upward for both gate forms, and it is largest at 350M for both. The combined arm with both primitives is the best arm at every scale. The combo2 arm (sink logit plus projection gate) has the lowest validation loss at 10M (4.1897 vs. 4.2119 baseline), 50M (3.5266 vs. 3.5399), 124M (3.3604 vs. 3.3710), and 350M (3.0210 vs. 3.0338).
Injection Experiments Reveal the Blind Spots
The third finding uses causal injection experiments to confirm that gates remove interference and to identify the characteristic weakness of each gate form. The procedure is straightforward: at evaluation time, for each head, identify its quiet reads (those whose norm falls below a per-head cutoff), add a corruption vector scaled to the head's median read norm and multiplied by a dose epsilon, and measure the increase in validation loss.
The structured junk condition uses a value read transplanted from a random position in another sequence. This junk has the statistical shape of a real read but no relation to the current context. At 124M, the baseline degrades by 0.11 nats at epsilon=0.4 and by 2.6 nats at epsilon=0.8 under structured junk. The sink logit, projection gate, and combined arms suffer between a tenth and a third of the baseline's damage at epsilon=0.4. At epsilon=0.8, the projection gate and sink logit suffer a tenth of the baseline's damage. Gaussian junk of the same norm costs the baseline only 0.035 nats at epsilon=0.4, a fourteenfold difference from structured junk. Noise with no feature structure is largely averaged away by aggregation, whereas a transplanted read carries feature directions that downstream layers respond to.
Each gate form has a blind spot. The norm gate is blind to magnitude: it attenuates reads based on their norm, so large junk that happens to have a high norm passes through. At 124M under structured junk, the norm gate loses 4.4 nats at epsilon=0.8, worse than the baseline's 2.6 nats. At epsilon=1.6, it loses 6.8 nats against the baseline's 4.9. The projection gate is blind to direction: it attenuates reads based on a learned linear function of their content, so junk injected along the gate's own learned direction passes through. At 124M, junk along a random direction at epsilon=0.8 costs the projection gate 0.29 nats, but junk along its own gate direction costs 4.8 nats, seventeen times as much.
The norm gate's weakness at high doses explains a paradox in the literature: some studies find that norm-based gates help, others find they hurt, and the difference often comes down to the noise regime. In normal training, the norm gate works because quiet reads tend to be interference and high-norm reads tend to be signal. Under adversarial or heavy-tailed noise, this heuristic breaks down. The projection gate is more robust because it can learn which directions correspond to interference, though it has its own blind spot along that direction.
The Design Corollary
The paper derives a concrete recommendation from the three findings. The lowest-loss arm at every scale is the one with both primitives built in: a learned per-head sink logit for exact abstention, and a projection gate on each value for noise filtering. The parameter cost is negligible: H parameters per layer for the sink logit and H(d+1) parameters per layer for the projection gate, where H is the number of heads and d is the head dimension. At 124M, this is under 0.01% of total parameters. The gated read can be computed once and stored in the key-value cache, so none of the gates changes the cost of generation.
The downstream evaluations at 124M are directionally consistent with the loss results. On LAMBADA, the combo arm reaches 0.186 accuracy versus 0.182 for the baseline. On HellaSwag, the gains are within noise. On WikiText-2 perplexity, the combo arm reaches 56.8 versus 58.1 for the baseline. The gated arms improve most on LAMBADA targets that appear within 32 tokens, suggesting the filtering helps with short-range retrieval from the context.
A survey of pretrained models confirms that the interference being filtered is present in ordinary transformers. Across GPT-2 (124M to 1.5B) and Pythia (70M to 2.8B), the blind-mass ratio (the norm of attention-weighted reads that carry no target feature, divided by the norm of reads that do) stays between 1.0 and 1.75 with no trend toward norm-aware routing as models grow. The routing mechanism does not learn to route around noise on its own. The value pathway needs an explicit filter.
Limitations and What Comes Next
The 350M results are single seeds, which limits confidence in the exact magnitudes at that scale, though the trend from 10M through 124M is established with three seeds per arm. All models are trained on one corpus (FineWeb-Edu) at modest token budgets, ranging from 0.2B tokens at 10M to 5.8B at 350M. The effects are on the order of a few thousandths of a nat, which is within the regime where seed noise matters, and every comparison in the paper is a paired per-seed difference to control for this. There is no transfer testing beyond 350M and no throughput measurements, though the authors note that the parameter cost is negligible and the key-value cache is unchanged.
Three open questions remain. First, does the filtering benefit continue past 350M, or does it plateau the way abstention does? The extrapolation from the existing data suggests it keeps growing, but the data stops at 350M. Second, by what mechanism does routing reorganize around a gate? The exposure accounting shows that gated models redirect mass away from quiet reads, but the dynamics of how this reorganization happens during training are not yet understood. Third, does the blind-spot principle apply to query-side and key-side gates, not just value-side gates? The theoretical argument is general: any gate that makes decisions along one variable is blind to variation along orthogonal variables, but the empirical test has not been run for other gate placements.
For practitioners, the message is clear. If you are adding value gating to a transformer, do not treat it as a single intervention with a single explanation. It supplies two things, and the one that matters depends on your model's scale. At small scales, the zero option dominates. At large scales, noise filtering dominates. The best design includes both, implemented as a learned sink logit and a projection gate, at negligible cost. The prior disagreement in the literature was not about whether gates help. It was about which of two different things they were doing.