Deriving text embeddings directly from frozen large language models has emerged as a promising direction, yet existing methods focus almost exclusively on English, leaving multilingual settings largely unexplored. This work presents the first systematic study of multilingual training-free text embeddings in LLMs.
Analysis reveals a language-conditioned embedding geometry in which semantically equivalent sentences remain misaligned across languages. The hypothesis that language-specific variation can be disentangled from semantic content to improve embedding quality motivates the proposed intervention.
Paired-Residual Subspace Removal (PRSR)
PRSR is a training-free inference-time method that steers LLM representations toward a more language-invariant semantic space. The procedure constructs paired residuals from multilingual parallel sentences to capture language-specific deviations. Layer-wise low-rank subspaces are estimated from these residuals, and the corresponding components are suppressed in the last-token hidden states.
By removing language-directed variation while preserving semantic content, the method improves embedding quality without requiring model retraining. The intervention operates at inference time, making it applicable to any frozen LLM.
Empirical evaluation
More than 200 monolingual evaluations across 18 languages were conducted using MTEB and MMTEB, alongside cross-lingual bitext mining between English and over 200 other languages. Empirical results demonstrate that PRSR consistently improves the performance of existing training-free baselines in both monolingual and cross-lingual settings across different LLMs.
The method addresses a gap in multilingual embedding research and provides a practical approach for improving cross-lingual representation quality.