The Neural Thickets hypothesis and its geometric claim
The Neural Thickets hypothesis, introduced in prior work, posits that pretrained language models lie in dense neighborhoods of task-specific experts in weight space. According to this view, random Gaussian perturbations to the model's weights move the parameters into regions that specialize on particular tasks. The proposed mechanism relies on the assumption that the model's pretrained parameters are near flat optima, such that even modest perturbations can land in basins corresponding to different tasks. Empirical results appeared to show that perturbing across all weight dimensions yields gains on reasoning benchmarks, reinforcing the idea that weight-space neighborhoods contain task-relevant structure.
However, the authors here reproduce the reported gains and then question the necessity of this geometric interpretation. Their core observation is that the improvements attributed to full-weight perturbations can be achieved through much simpler means. Perturbing a single decoder block, for instance, already recovers much of the improvement. Even more strikingly, adding a random bias to the output logits of a frozen model matches or exceeds full-weight perturbations on most reasoning tasks. These findings suggest that the original hypothesis, while intriguing, may not be the primary driver of the reported benefits.
From weight perturbations to output diversity: the core finding
The pivotal finding of this paper is that the gains from weight perturbation do not require touching the weights at all. When the model is kept completely frozen and a random bias vector is added to its output logits, performance on reasoning tasks remains comparable to that of full-weight Gaussian perturbation. This result directly challenges the assumption that weight-space geometry is essential. The authors also show that perturbing just one decoder block suffices to recover much of the observed improvement, further narrowing the scope of what is actually needed.
The alternative explanation that emerges focuses on output diversity. By introducing variation into the model's predictions—whether through weight perturbations, single-block perturbations, or logit biases—the model produces a broader range of outputs. When combined with selection on a validation set, this diversity becomes a resource rather than a side effect. The paper argues that what looks like "finding experts in weight space" is in fact the result of generating diverse predictions and selecting the best ones.
Perturb-Select-Vote: a formal alternative framework
The authors formalize their alternative as Perturb-Select-Vote (PSV). The procedure has three stages:
- Perturb: generate multiple perturbed predictors. In the weight-based variant, this means applying random Gaussian perturbations to some or all of the model's parameters. In the logit-based variant, this means adding a random bias to the output logits of a frozen model.
- Select: evaluate each perturbed predictor on a small validation set and keep the high-performing members. The selection criterion can be based on loss, accuracy, or any task-relevant metric.
- Vote: aggregate the predictions of the selected predictors, typically by averaging logits or taking a majority vote.
PSV is general: it does not require retraining, and it works with any frozen model. The authors show that the weight-perturbation results reported in prior work can be reproduced essentially as a special case of PSV, where the perturbation step happens in weight space rather than at the output level.
This interpretation connects Neural Thickets to established inference-time methods. Sampling methods generate diverse predictions by drawing from the model's stochastic behavior. Self-consistency selects the most consistent answer across multiple generations. Verifier-guided selection uses an external validator to rank predictions. PSV subsumes all of these as instances of perturb-then-select-then-aggregate, with the key difference being that the perturbation step can operate on the output logits alone.
First-order analysis: weight perturbations and logit shifts
The authors provide a first-order analysis showing that small weight perturbations act primarily through their induced logit shifts. Using a Taylor expansion of the model's output with respect to the weights, they demonstrate that the dominant effect of a perturbation is a linear shift in the logits, scaled by the gradient of the output with respect to the weights. Higher-order terms are negligible for small perturbations. This means that, to first order, perturbing weights is nearly equivalent to directly adding a bias to the logits—precisely the logit-PSV procedure the authors propose.
The analysis extends naturally to the vision domain. The authors experiment with domain shift scenarios, including Earth observation imagery, where the base model was pretrained on natural images. Full-weight perturbations provide little additional benefit beyond what logit PSV already achieves. In these settings, the diversity introduced by a logit bias is sufficient to improve accuracy under distribution shift, while the extra complexity of perturbing all weights yields diminishing returns.
Vision experiments: domain shift and Earth observation
In the vision portion of the study, the authors test logit PSV on domain shift benchmarks. The base model is a vision backbone pretrained on natural images, and the task involves classifying examples from a satellite imagery dataset (Earth observation). The results show that logit PSV substantially improves accuracy compared to the unperturbed model, and that the improvement is nearly on par with full-weight perturbation ensembles. However, full-weight perturbations provide little additional benefit over logit PSV alone. This pattern mirrors the language findings: the key ingredient is output diversity and selection, not the full weight-space geometry.
The authors attribute this to the nature of domain shift. When the input distribution changes, the model's logits shift in ways that can be partially corrected by introducing diverse perturbations and selecting the most appropriate ones. Perturbing the weights further does not provide a meaningful additional signal because the logit-level perturbation already captures the dominant effect.
Practical implications: logit PSV as a training-free alternative
The practical takeaway from this work is that logit PSV offers a simple, training-free alternative to full-weight perturbation ensembles. Because the perturbation operates on the output logits, no gradient computation, no weight update, and no additional training is required. The model remains frozen throughout. This makes PSV attractive for deployment scenarios where compute resources are limited or where a model cannot be modified after training.
For language reasoning tasks, logit PSV matches or exceeds full-weight perturbations while requiring only a single forward pass with added bias per perturbed sample. For vision under domain shift, logit PSV provides substantial gains over the baseline with minimal computational overhead. The authors note that the validation set used for selection can be very small—perhaps a few dozen examples—without significantly affecting the quality of the aggregated prediction.
The broader significance of this work is that it reinterprets previously reported gains through a simpler lens. The Neural Thickets hypothesis, while geometrically appealing, may not be the correct explanation for why perturbation-based ensembling works. Instead, the mechanism appears to be output diversity, validation-based selection, and voting—principles that are well-established in inference-time scaling but now formalized as a unified procedure. Logit PSV makes these principles accessible with a fully frozen model, opening the door to practical ensemble methods that were previously considered too expensive or too tied to weight-space geometry.