Prompted embedding models have recently received increasing attention, particularly for retrieval, where detailed retrieval instructions are provided as part of the retrieval prompt. Several new datasets and studies have examined this setting, showing that the current embedding models often struggle to follow such instructions reliably. In this paper, we study the mechanism of how instructions actually affect the representations of retrieval queries in asymmetric retrieval tasks. We show that models can fail to follow even simple task instructions when query-side distractors are included in the evaluation. We hypothesize that this behavior is driven by the training setup of current embedding models and their evaluation, and show that fine-tuning with added query-side distractors leads to substantial improvements, with minimal effect on other tasks.
Why Retrieval Instructions Matter for Embedding Retrieval
Text embeddings have become a foundational ingredient for search, clustering, and classification systems. Modern models are trained to produce vector representations where semantic proximity correlates with functional similarity. In asymmetric retrieval settings, only the query carries a task-specific instruction; candidate documents are embedded independently, which permits a shared index to serve multiple tasks. This design choice has driven the adoption of instructed embedding models across domains, from question answering to multilingual bitext mining. Yet the promise of a single model adapting to many tasks collides with a persistent problem: instructions are frequently ignored or only weakly reflected in the resulting vectors.
The paper begins by framing the mismatch between training and evaluation. Popular benchmarks such as MTEB evaluate models on task categories in isolation: both queries and candidate sets are constructed for a single task, and the candidate pool rarely contains examples that are semantically related to the query without satisfying the task instruction. This evaluation protocol provides limited insight into how instructions transform query embeddings and alter retrieval behavior. The authors argue that this setting overestimates genuine instruction sensitivity because it does not expose models to the kind of candidate pools encountered in large-scale retrieval, where semantically similar but task-irrelevant texts are the norm.
Several related studies have examined instruction following in embedding models. Instruct datasets have been introduced covering everything from preferred source types to expert healthcare criteria, yet these instructions are often complex and query-specific, making systematic analysis of task-level effects difficult. Other work has investigated the geometric effects of inputs on representations, showing that interpretable axes exist in embedding space, but the connection between instruction content and representational change remains incompletely understood. The contribution of this paper is to move beyond benchmark scores and analyze how roughly 50 diverse instructions affect query representations with structural measures that quantify the shifts instructions introduce. The authors additionally identify a core weakness: standard fine-tuning and evaluation setups lack query-side negatives, which are exactly the kind of texts that break instruction following in practice.
The Distractor Problem: How Query-Side Text Breaks Instruction Following
The central empirical finding of the paper is that current models fail to follow retrieval instructions when the candidate pool includes semantically similar query-side distractors. The authors generate these distractors by prompting a language model to paraphrase each query while preserving its meaning. These paraphrase queries serve as hard negatives: they are related to the query by plain semantics but do not fulfill the task instruction. When such distractors are added to the retrieval pool, performance collapses for the baseline models across both the ARCChallenge question-answering benchmark and the Tatoeba translation-task benchmark.
To quantify this effect, the authors introduce three structural metrics. Similarity increase measures whether adding an instruction raises the cosine similarity between the query and its correct target. Displacement computes the cosine distance between the query embedding and the instruction-affected query embedding, indicating how much the instruction moves the representation. Angulation measures the cosine similarity between the direction of instruction-induced movement and the direction toward the correct target, with values near 1 indicating that movement was predominantly toward the target and values near 0 indicating orthogonal movement. These three metrics together diagnose whether an instruction is merely shifting the query vector without guiding it toward the correct target.
The results are striking. On ARCChallenge, low displacement correlates with high evaluation scores, and relevant instructions are not distinguishable from other instructions at the top end of the scores. Similarity improvement does not correlate with performance for relevant instructions; the majority of retrieval instructions land on the negative side, meaning they actually decrease similarity to the correct target. Angulation shows that most high-scoring instructions do not move along the query-target line. On Tatoeba, the pattern is somewhat better: relevant instructions display almost full separation from other instructions under all three metrics, but the relationship between low displacement and high score is still pronouncedly different between relevant and other instructions. Across all models, the cross-validated ROC-AUC values for separating relevant from other instructions are moderate at best, confirming that standard evaluation scores are poor proxies for whether instructions are actually being followed.
The distractor experiments make the problem concrete. Adding a single paraphrase distractor per query completely destroys Recall@1 performance on ARCChallenge for all models. On Tatoeba, two models almost perfectly retain performance with distractors, while the best-scoring instructions are relevant. Figures in the paper illustrate the effect: in the Tatoeba case, the position of each query–instruction pair relative to the query, distractor, and target shows that models that retain performance are those that move away from the query and toward the target rather than toward the distractor. The takeaway is clear: models that react to instructions exist, but the majority fail to do so when semantically similar but task-irrelevant texts are present.
The Training-Evaluation Gap: Why Standard Benchmarks Miss the Issue
The authors hypothesize that the root cause of inadequate instruction following is the training setup of current embedding models and their evaluation. Standard contrastive objectives use in-batch negatives and target-side hard negatives mined from the corpus, but they do not include query-side negatives—texts that are related to the query by semantics but should not be retrieved under the given task instruction. Without such negatives during training, the model has little incentive to repel query-side distractors. Similarly, standard evaluation isolates tasks: both queries and candidate sets are constructed for a single task, so the candidate pool rarely contains the kind of query-side semantically similar texts that would challenge instruction following. The paper demonstrates that fine-tuning models with added query-side distractors substantially improves robustness, which supports the hypothesis that the gap is tractable.
This hypothesis is further supported by the language-family difference observed between the two benchmarks. Tatoeba, which comprises translation pairs across eight languages, shows more desirable behavior than ARCChallenge. The authors conjecture that mixing tasks during fine-tuning—exposure to slightly more varied negatives—improves language sensitivity and explains the better results on Tatoeba. In contrast, ARCChallenge’s retrieval setting, with its focus on a single task and its test-only evaluation, provides fewer opportunities for the model to learn to distinguish task-relevant from task-irrelevant query-side texts.
The paper also examines structural metrics after fine-tuning. In the SQuAD-fine-tuned model, similarity improvement increases on average: after fine-tuning, all relevant instructions increase similarity to the answer more than before fine-tuning, not only the fine-tuning instruction. The effect is opposite for Tatoeba: similarity improvement decreases slightly in all languages, but less for relevant instructions. Displacement grows in both fine-tuned models, and both models also increase in angulation, especially on Tatoeba where most increase is seen in the prompt where the language is explicitly mentioned—a desirable outcome. Out-of-domain evaluation on the full MTEB benchmark shows that fine-tuning for distractors does not destroy performance on other tasks; absolute differences across categories are small. This finding confirms that the proposed remedy improves instruction-following robustness without substantial collateral damage to the model's broader capabilities.
Structural Metrics: Displacement, Similarity Increase, and Angulation
The paper's introduction of displacement, similarity increase, and angulation provides a framework for diagnosing instruction effects that goes beyond simple accuracy scores. Similarity increase is the most intuitive metric: if the value is greater than zero, adding the instruction increased the overall similarity between the query and the correct target; otherwise, it decreased it. However, the authors note that naively moving toward the correct target may also mean moving toward incorrect targets, which is why the other two metrics are necessary.
Displacement measures the magnitude of the effect adding instructions has. Values close to zero indicate high similarity between the original query embedding and the instruction-affected embedding, meaning low displacement. If displacement is too large, the movement may be detrimental. The authors also calculate displacement with respect to the targets and distractors separately, which reveals whether the instruction moves the query toward the correct target or toward the distractor.
Angulation accounts for the limitation of displacement by measuring the cosine similarity between the vector from the original query to the instruction-affected query and the vector from the original query to the correct target. A value of 1 indicates that all movement was in the direction toward the target, while a value of 0 indicates orthogonal movement. Together, these three metrics can diagnose whether an instruction is effectively guiding the query embedding toward the correct target or merely moving it without direction.
The cross-validated ROC-AUC results, reported in Table 3 of the paper, quantify how well each structural metric separates relevant from other instructions. On ARCChallenge, the values are lower than on Tatoeba, meaning relevant instructions have more distinct behavior under the structural metrics on Tatoeba than on ARCChallenge. The Spearman correlations between evaluation scores and the structural metrics display the dominance of similarity improvement on Tatoeba and the negative relationship between displacement and evaluation score for both datasets. These results confirm that no single metric provides a complete picture, and that the combination of all three is necessary for meaningful diagnosis.
Fine-Tuning With Distractors: A Targeted Remedy
The paper's proposed remedy is to fine-tune embedding models with query-side distractors included in the training data. The authors fine-tune the Qwen/Qwen3-Embedding-0.6B model separately for question-answering and bitext mining, using the SWIFT framework. Each training example consists of a query, its retrieval target, and a distractor generated by paraphrasing the query while preserving its meaning. To avoid catastrophic forgetting, the authors fine-tune jointly on retrieval tasks and semantic textual similarity (STS), where the distractor serves as a positive target for STS but a query-side negative for the retrieval tasks. For 50% of the training examples, one negative is replaced by the wrong-task target, further regularizing the model against overfitting to the retrieval setup.
The fine-tuning results are compelling. On SQuAD development, the fine-tuned model's Recall@1 under paraphrase distractors improves from 0.014 (baseline) to 0.470, a more than 30-fold increase. On Tatoeba, the fine-tuned model's Recall@1 under paraphrase distractors improves from 0.102 to 0.569. For random distractors, the fine-tuned model's performance is nearly unaffected, indicating that the improvement is specific to semantically similar query-side negatives rather than a general increase in candidate-set size. Critically, the Normal results—standard retrieval performance without added distractors—remain close to the baseline, confirming that the fine-tuning does not sacrifice baseline retrieval quality.
The paper also analyzes the structural effects of fine-tuning. Similarity improvement increases for the SQuAD-fine-tuned model across all relevant instructions; for the Tatoeba-fine-tuned model, similarity improvement decreases slightly in all languages but less for relevant instructions. Displacement grows in both fine-tuned models, and both increase in angulation, especially on Tatoeba where the increase is concentrated in prompts that explicitly mention the target language. These changes align with the characteristics identified in Section 4 as beneficial: larger instruction effects and better performance retention when distractors are added.
Results: Improved Robustness Without Sacrificing Baseline Performance
The test-set results, reported in Table 6 for the Qwen/Qwen3-Embedding-0.6B model and its fine-tuned variants, confirm the robustness improvement. For the baseline model, paraphrase distractors almost completely destroy Recall@1 performance: despite being instructed to retrieve the target, the model typically retrieves the query-side distractor instead. Random distractors cause only a moderate drop, indicating that simply increasing the candidate set or adding unrelated queries does not substantially mislead the model. The fine-tuned models show different behavior: their performance is nearly unaffected by random distractors, and although paraphrase distractors still reduce performance, the degradation is substantially smaller than for the baseline. This suggests that the model can learn to follow task-specific instructions from a relatively small amount of fine-tuning data.
The fine-tuning does not substantially degrade standard retrieval. The Normal results in Table 6 remain close to the baseline for both SQuAD and Tatoeba. In Appendix H, the authors show that the improved robustness to paraphrase distractors is not due to task-specific fine-tuning alone but specifically to the addition of query-side negatives. When models are fine-tuned on the same data using the same setup but excluding query-side negatives, performance under paraphrase distractors remains flat, and the gap between the Normal and Random evaluation settings remains. This confirms that it is the query-side negative examples during training that produce the improvement.
Out-of-domain performance is evaluated on the full MTEB English v2 benchmark. Figure 5 shows performance on MMTEB benchmark categories for the fine-tuned models, with absolute differences small enough that the fine-tuning setup does not affect overall performance. The full MTEB results in Appendix I confirm that while fine-tuning for distractors has effects on results on other benchmarks, the differences are small and do not destroy performance on other tasks. The SQuAD-fine-tuned model evaluated on ARCChallenge shows slight improvement in the paraphrase-distracted evaluation for Recall@1 and Recall@5, even though the baseline performance was already low. These findings collectively demonstrate that the proposed fine-tuning strategy improves instruction-following robustness without substantial collateral damage.
What This Means for Retrieval System Developers
For practitioners building retrieval systems with instructed embedding models, the paper's findings have several immediate implications. First, standard benchmark scores should be interpreted with caution: a model that scores well on MTEB or similar benchmarks may still fail to follow instructions when the candidate pool contains semantically similar query-side texts, which is the norm in large-scale retrieval. Second, the structural metrics introduced in the paper—displacement, similarity increase, and angulation—provide a diagnostic framework for assessing whether a given model and instruction setup is likely to be robust in practice. Third, and most importantly, fine-tuning with query-side distractors is a low-cost, high-impact remedy. The authors demonstrate that a single round of fine-tuning on a few thousand examples, with distractors comprising paraphrases of the training queries, substantially improves instruction-following robustness while leaving baseline retrieval quality essentially unchanged.
The workflow for applying this remedy is straightforward. Collect a training set of queries with their correct targets, generate paraphrase distractors for each query (the paper uses GPT-4o-mini for this), and fine-tune the embedding model on the combined retrieval and STS tasks with the described joint setup. The authors use 50% of training examples with one negative replaced by the wrong-task target, a simple but effective regularization technique. The fine-tuning can be completed in a few hours on a single GPU for the model sizes examined in the paper. After fine-tuning, the model should be evaluated not only on standard benchmarks but also with query-side distractors added to the retrieval pool, to confirm that the desired robustness improvement has been achieved.
For researchers, the paper opens several directions for future work. The analysis focuses on retrieval and bitext-mining tasks; extending the framework to other common embedding tasks such as clustering would be valuable. The study examines eight prominent text embedding models, but broader model coverage would strengthen the generalizability of the findings. The synthetic distractor data is generated using a single model, and the authors acknowledge that the choice of generation model may have limited but non-zero effect on the results. Future work could ablate over different generation models and investigate whether the improved robustness transfers across model families. Finally, the paper's methodology for assessing inadequate instruction-following and the fine-tuning framework to address it are not tied to any specific architecture, making them applicable to any model exhibiting similar characteristics.
Read the paper on arXiv