Positional Encoding May Be Hurting Your Transformer
Transformers are everywhere, and positional encoding is considered foundational. Every major production model, from Llama to DeepSeek, uses RoPE (Rotary Position Embeddings) to tell the model where each token sits in a sequence. The assumption is obvious: without explicit position information, a model cannot distinguish "the cat sat on the mat" from "the mat sat on the cat." But what if the position signal is doing more harm than good for certain generalization behaviors?
This paper from Daniel Henrik Nevermann and Claudius Gros at Goethe University Frankfurt tackles a specific but important failure mode: distance generalization. Unlike the better-studied problem of length generalization, where models must extrapolate to sequences longer than those seen in training, distance generalization asks whether a model can handle different inter-token distances within a fixed context window. If a model is trained to copy tokens that are 15-25 positions apart, can it copy tokens that are 30 or 40 positions apart, while the total sequence length stays the same?
The distinction matters because length generalization conflates two separate problems: the model might fail because it has never seen those positions, or because it has never seen those token dependencies. Distance generalization isolates the second problem. The authors construct two synthetic delay copy tasks, train causal decoder-only transformers (8 layers, 8 heads, 512 dimensions) with three positional encoding schemes, and find a result that challenges conventional wisdom: NoPE (no positional encoding) consistently outperforms both RoPE and ALiBi on distance generalization.
Why Distance Generalization Is Distinct From Length Generalization
In length generalization, a model trained on sequences of length 128 must handle sequences of length 256 at inference. This tests whether the model can generalize to unseen positions. In distance generalization, the context length stays fixed, but the gap between a source token and its recall position changes. The model must generalize to unseen token dependencies without ever encountering unseen positions.
This separation is methodologically important. When a model fails at length generalization, researchers cannot tell whether the failure stems from position encoding limitations or from an inability to handle longer-range dependencies. Distance generalization removes the position confound and tests the dependency mechanism directly.
The authors use a task switching framework where sequences contain interleaved tasks. The delay copy task (C) requires copying m tokens from a source position to a recall position separated by a variable delay d_C. The selective copy task (S) copies only even-numbered tokens, adding a filtering requirement. Random tokens (R) are inserted between task pairs to prevent shortcut learning. Training uses next-token prediction with teacher forcing, and evaluation measures accuracy per delay distance, probing both in-distribution (within the training delay range) and out-of-distribution (beyond it) performance.
RoPE Is the Worst, and NoPE Is the Best
The most striking finding is in Figure 2. When trained on delay distances between 15 and 25, all three encoding schemes achieve high in-distribution accuracy. But outside the training range, performance diverges sharply. RoPE degrades fastest, ALiBi is intermediate, and NoPE maintains the highest accuracy at unseen distances.
This is counterintuitive because RoPE explicitly encodes relative positions through rotary rotations, which should in principle give it the information needed to handle arbitrary distances. ALiBi adds a linear bias to attention scores that decreases with distance, providing a recency signal. NoPE uses no positional information at all, relying solely on causal attention masking.
The authors note this aligns with findings from Kazemnejad et al. (2023) on length generalization, where NoPE also outperformed explicit encodings. The explanation they offer is that causal transformers can dynamically learn to encode relative positions from the attention pattern itself, and that explicit positional encodings may interfere with this learned mechanism rather than helping it.
However, the authors add an important caveat from their appendix: for very small models, explicit positional encoding is actually necessary. NoPE breaks down at small model sizes, suggesting the model needs sufficient capacity to learn positional representations from scratch. The question of where exactly this threshold lies, and whether it depends on task complexity, remains open.
Data Diversity Helps Absolutely but Diminishes Relatively
The authors vary the number of delay distances seen during training by setting d_min = 0 and d_max in {5, 10, 20, 30, 50}, giving N_d = d_max distinct distances. As training diversity increases, two things happen simultaneously.
First, absolute out-of-distribution performance improves. Models trained on a wider range of distances generalize better to unseen distances, as shown in the left panels of Figure 3. More training distances means the model encounters a broader range of dependency structures, which helps it generalize.
Second, the ratio P = P_out / P_in, measuring relative generalization (out-of-distribution accuracy divided by in-distribution accuracy), decreases. The authors define this as diminishing returns: increasing data diversity produces sub-linear gains in generalization relative to the increase in training diversity. The reason is that models reach an accuracy floor before the end of the evaluation range. Adding more training distances helps, but each additional distance contributes less to the overall generalization capability.
This has practical implications. If you are designing training data for a transformer and want it to generalize to unseen dependency distances, there is a point of diminishing returns where collecting more diverse training examples yields smaller and smaller relative improvements. The absolute gains still matter, but the cost-benefit calculation shifts.
The authors also raise an open question: is there a minimum data diversity threshold below which generalization is impossible? Their data does not rule this out, and they note that Izzo et al. (2025) have found such thresholds for length generalization. Whether distance generalization has an analogous phase transition deserves further investigation.
Transfer Learning Is Asymmetric and Sometimes Destructive
The most complex findings concern transfer learning between the full copy task (C) and selective copy task (S). The authors train models on a joint dataset containing both tasks, with different delay distance ranges for each. For example, C is trained on distances [15, 25] while S is trained on [25, 35]. They then test whether the presence of one task helps or hurts generalization on the other.
The results depend heavily on the positional encoding scheme and the distance separation between tasks.
For RoPE, transfer learning is mostly negative. The presence of the second task acts as a distractor, causing out-of-distribution performance on C and S to decay faster than when each task is trained alone. This contrasts with Cai et al. (2025), who found mostly positive transfer learning for length generalization. The difference may be that distance generalization creates a different failure mode where task interference is more likely.
For ALiBi and NoPE, the pattern is more nuanced. At moderate out-of-distribution distances, transfer learning helps. But at larger distances, the effect reverses and becomes destructive. Most strikingly, when the task distance ranges are well-separated (C at [15, 25], S at [45, 55]), the destructive effects appear even within C's own training region. The presence of the S task hurts C's performance at distances C was trained on.
The authors propose an explanation: transfer from a more complex task (S, which requires filtering) to a simpler one (C, which copies everything) may be favored, while transfer in the opposite direction is less effective or harmful. They also observe an asymmetry in the direction of constructive transfer: for NoPE and ALiBi, S-to-C transfer improves C's generalization around S's training range, but C-to-S transfer does not produce analogous benefits.
A key advantage of RoPE in this context is that while transfer learning is weak, destructive effects within the original training region are not observed. For NoPE and ALiBi, the distractor task can degrade performance even on distances the model was explicitly trained on, a failure mode that RoPE avoids.
The Practical Implications for Model Design
The findings raise uncomfortable questions about the ubiquitous use of RoPE. If NoPE consistently outperforms RoPE on distance generalization, and this aligns with similar findings on length generalization, then the position encoding used in virtually every production model may be a net negative for certain generalization behaviors.
Of course, the experiments use synthetic tasks, small models, and controlled settings. Whether these results transfer to large-scale language models on natural language tasks is genuinely unknown. The authors explicitly state that distance generalization needs to be tested in pretrained models, and that follow-up studies should investigate more diverse tasks with in-depth circuit and attention analyses.
But the core finding is clear enough to warrant investigation. Causal attention masking provides an implicit position signal: tokens can only attend to earlier positions, which means the attention pattern itself encodes relative ordering. When explicit positional encodings are added on top of this, they may constrain the model's learned representations rather than enhancing them.
For practitioners, the takeaway is that positional encoding is not a free lunch. It provides in-distribution benefits, but may come at the cost of out-of-distribution generalization. The right encoding depends on the deployment context: if the model will only see in-distribution positions, RoPE is fine. If it needs to generalize to unseen distances or lengths, NoPE or ALiBi may be better choices.
Open Questions
The authors identify several directions for future work. First, whether there exists a minimum data diversity threshold for distance generalization, analogous to the lower bounds found for length generalization. Second, circuit-level analysis of how transformers with and without positional encoding represent and compute distances. Third, testing on pretrained models rather than training from scratch. Fourth, investigating the asymmetry in transfer learning directionality and whether it generalizes to other task pairs. Fifth, determining the model-size threshold at which NoPE becomes viable, since it breaks down at small scales.