Long-context language models can access evidence buried deep within extensive documents, but accessing it is not the same as using it. A paper submitted to arXiv on September 22, 2026, identifies a previously under-examined problem in long-context modeling. When a model struggles to retrieve distant evidence, the cause is not always positional distance. More often, the problem is cumulative competition from abundant, task-irrelevant context that sits near the query position. The authors call this the Proximity Trap, and they propose LYRA, a lightweight mechanism that reshapes how attention scores are computed to resist this interference.
The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence comes from Xiaoyu Yang, Jie Lu, Wei Duan, and En Yu at the Australian Artificial Intelligence Institute and the University of Technology Sydney. The paper introduces LYRA (Long-context heavY-tailed Relevance Alignment), a t-distributed directional matching mechanism, alongside ProxBench, a new benchmark designed to measure exactly this failure mode. Experiments across LongBench-v2, RULER, and LongBench show consistent improvements, with LYRA achieving the best overall score on LongBench-v2 at 36.72 percent versus 32.21 percent for the baseline.
The Proximity Trap: Distance Is Not the Real Problem
Modern LLMs encode relative positions through rotary positional embedding, making query-key attention scores explicitly dependent on distance. This positional bias is a reasonable inductive bias, since nearby tokens often provide useful information for the current prediction. The problem is that positional proximity does not determine task relevance. Essential evidence may lie far from the query, while nearby content may provide little information for the required prediction.
Prior research on long-context models has focused almost exclusively on overcoming distance. The assumption has been that if a model can attend to distant tokens effectively, it will use them. The authors challenge this assumption with a counter-intuitive observation. On LongBench-v2, progressively masking the context tokens nearest to the query consistently improves overall accuracy compared with the unmasked baseline, with an even more pronounced improvement on Long In-context Learning. Removing accessible context improves performance rather than degrading it. This suggests that the difficulty of using distant evidence cannot be attributed to distance alone.
Through two complementary experiments on Qwen3-8B, the authors establish the core finding. First, layer-wise attention analysis shows that the model can precisely identify distant evidence among ordinary tokens at the same distant location. From approximately layer 19 onward, attention to distant evidence increases sharply, while its neighborhood remains below the uniform baseline. However, proximal background also receives substantial attention. Although its per-token attention is generally lower than that of distant evidence, its abundance produces sufficient cumulative attention mass to overshadow the evidence.
The second experiment compares two interventions. Moving distant evidence closer does not consistently improve performance, as the results span all four quadrants of a paired scatter plot. In contrast, attenuating proximal background scores while keeping positions fixed produces a more consistent improvement in both evidence attention and answer confidence. The conclusion is stark: shortening the evidence distance does not remove competition from abundant proximal background. Attenuating that background directly weakens the competition.
How Proximal Background Overshadows Distant Evidence
The mechanism behind the Proximity Trap can be understood through the mathematics of softmax normalization. When RoPE is applied, the query-key score depends on the relative offset between positions. Distance changes the directional match between query and key across frequencies rather than reducing their vector lengths. For a maximally aligned query-key pair, the score attenuates quadratically with distance.
However, score attenuation alone does not determine evidence attention. Consider one distant evidence token with QK score s_E and M proximal background tokens, each with QK score s_B. Their softmax competition gives the evidence attention as alpha_E = 1 / (1 + M * exp(s_B - s_E)). The evidence receives less than half the attention weight whenever s_E - s_B < log M. Even a moderate reduction in the evidence score can be exponentially amplified by softmax normalization, while the cumulative suppression from proximal background grows linearly with the number of background tokens.
This formulation explains both experimental observations. The model attends to distant evidence, but abundant proximal background can collectively overshadow it through the multiplicative factor M. Attenuating proximal background directly reduces this factor, restoring evidence attention. Moving the evidence closer does not change M, so it leaves the fundamental competition unresolved.
LYRA: t-Distributed Directional Matching
LYRA addresses the Proximity Trap by reshaping the QK scoring function. The core idea is to measure the directional agreement between the RoPE-transformed query and key, then apply a t-distributed transformation to reshape their score differences before softmax normalization. This design reduces misleading advantages held by weakly related proximal background while making strongly matched evidence more distinguishable.
Formally, LYRA first computes the cosine similarity between the RoPE-transformed query and key, yielding a value in the range negative one to one. It then applies the transformation phi_kappa(c) = (1 + c) / (1 + kappa * (1 - c)) - 1, where kappa controls the angular concentration parameter. When kappa equals zero, the transformation recovers the original cosine similarity. For positive kappa, the transformation is strictly increasing, preserving the original similarity ordering. Below a critical similarity threshold, the transformation compresses differences. Above it, the transformation amplifies differences.
This behavior is precisely what is needed to counter the Proximity Trap. When proximal background has a higher similarity than distant evidence, the transformation compresses the background advantage. When evidence retains a higher similarity, the transformation enlarges the evidence advantage against cumulative background competition. The relative weight of evidence is improved without introducing an explicit position-dependent increase for distant tokens.
Crucially, LYRA modifies only the query-key scoring stage of attention. RoPE, causal masking, softmax normalization, and value aggregation remain unchanged. The mechanism can be integrated into an existing attention layer by replacing its QK scoring function without modifying the remaining architecture. This makes LYRA a drop-in replacement that adds minimal computational overhead.
Experimental Results Across Context Lengths and Tasks
The evaluation uses Qwen3-8B as the base model, replacing the standard attention mechanism only in its final Transformer block with LYRA, then fine-tuning on LongAlign for one epoch. Experiments span three benchmarks.
On LongBench-v2, LYRA achieves the best overall performance at 36.72 percent, compared with 32.21 percent for the unmodified baseline and 33.82 percent for the previous strongest method, ProxyAttn. The improvement is consistent across all context-length splits: 47.22 percent on Short inputs versus 36.67 for the baseline, 29.63 on Medium, and 33.33 on Long versus 30.56. The leading performance on the Short split indicates that the improvement does not come at the cost of local context modeling.
On RULER, which evaluates the same controlled tasks at explicitly specified input lengths, LYRA achieves the best result at every evaluated context length from 8K to 128K. The scores are 96.33 at 8K, 94.19 at 16K, 93.14 at 32K, 85.39 at 64K, and 77.56 at 128K, yielding an average of 89.32 percent. The consistent advantage across the full range demonstrates that the effectiveness of LYRA is not restricted to a particular context range. The substantial improvement at intermediate and long context lengths, where relevant evidence faces increasingly strong competition from background tokens, is particularly notable.
On LongBench, which covers six task categories including single-document question answering, multi-document question answering, summarization, few-shot learning, code completion, and synthetic tasks, LYRA achieves an average of 50.06 percent, the best among all methods compared. The improvements are especially clear for multi-document question answering at 43.35 percent versus 37.17 for the previous best, summarization at 24.73 versus 16.66, and few-shot learning at 62.17 versus 36.20. These are precisely the tasks where evidence is distributed across long contexts and competes with substantial background information.
ProxBench: Measuring the Proximity Trap Under Control
Existing long-context benchmarks provide limited control over the proximal background context that competes with distant evidence. To address this, the authors introduce ProxBench, a controlled benchmark that fixes the query and distant evidence while progressively increasing the semantic similarity and relational ambiguity of proximal background context.
ProxBench organizes proximal perturbations into four progressively challenging levels. Level 1 introduces style-matched background that follows a similar syntactic structure but differs in entity, topic, and answer relation. Level 2 creates crossed bindings by placing the target entity and an access-code cue in the same sentence while associating the candidate value with another entity. Level 3 mixes the same relation for different entities with different attributes of the target entity. Level 4 introduces fine-grained perturbations involving subtypes, attributes, semantic roles, and value formats, producing background context that closely resembles the required evidence while remaining logically irrelevant.
On ProxBench, LYRA achieves the best average performance across all four difficulty levels when compared with Qwen3-8B, Llama3.1-8B, and GLM-4-9B. Although Llama performs slightly better at the easiest level, LYRA consistently achieves the strongest results at all subsequent levels, with the advantage becoming particularly clear under the most fine-grained perturbations. The performance of baseline models declines sharply when the proximal background shares entities, relations, attributes, or value formats with the target evidence. LYRA exhibits substantially greater robustness as the perturbation difficulty increases.
Limitations and What Comes Next
The paper has several limitations worth noting. LYRA is evaluated on a single base model, Qwen3-8B, and has not been tested on larger or differently architected models. The mechanism replaces attention scoring only in the final Transformer block, which means earlier layers still use conventional scoring. Whether applying LYRA across all layers would yield further gains remains open.
The t-distributed transformation introduces two hyperparameters, kappa and beta, that require tuning. The paper does not provide a systematic analysis of how these parameters interact with different model architectures or task types. The ProxBench benchmark, while well-designed, covers only a limited set of task types and perturbation patterns.
Perhaps the most important limitation is that the Proximity Trap is a phenomenon observed in specific input distributions. The severity of the problem depends on the ratio of proximal background to distant evidence in a given context. In contexts where background is sparse, the effect may be negligible. The practical value of LYRA therefore depends on the prevalence of long-context tasks where abundant task-irrelevant content sits near the query.
What This Means for Practitioners
For developers working with long-context LLMs, the paper offers a practical insight. The bottleneck in long-context understanding may not be how far the model can reach, but what it must compete with along the way. When building applications that rely on retrieving information from long documents, the presence of abundant nearby boilerplate, formatting, or example content can silently degrade the quality of evidence retrieval.
LYRA is available as a drop-in modification to the attention mechanism. The code is publicly available on the authors' project page. The implementation replaces only the QK scoring function, meaning it can be integrated into existing transformer architectures with minimal changes. The computational overhead is modest, as the transformation operates on the already-computed query and key representations.
From a broader research perspective, the paper reframes the long-context problem. Rather than asking how to help models attend to distant tokens, the more important question is what those distant tokens must compete with. The finding that masking proximal background improves performance suggests that context compression and filtering strategies should consider not just what to keep, but what to remove. The t-distributed transformation provides a principled way to implement this filtering at the attention level without modifying the model architecture.
Long-context models are increasingly used for document analysis, code review, and multi-turn conversation, all of which involve long stretches of task-irrelevant context near the current query position. The Proximity Trap is likely to affect these applications. LYRA offers a concrete solution that is architecture-agnostic, computationally efficient, and empirically validated across multiple benchmarks and task types.