When you type "2 kilometers" and "2000 meters" into a sentence embedding model, the cosine similarity between them should be nearly 1.0. After all, they denote the same physical quantity. A well-functioning semantic measurement space would also rank "1 meter" as closer to "105 centimeters" than to "15 kilometers." But what if the embedding model does the opposite? What if it thinks "2.5 meters" is more similar to "7.5 meters" than to "3 meters" simply because the strings share the character "5"?

Juri Opitz and Andrianos Michail at the University of Zurich set out to test exactly this. In their September 2026 paper, they evaluated 24 embedding models across four physical dimensions (length, mass, volume, time) and five numerical scales, asking a deceptively simple question: do embedding spaces reflect the objective structure of physical measurement? The answer, across every model tested, is no.

Why Physical Measurements Are a Perfect Testbed

Most NLP tasks involve fuzzy semantic judgments. Whether two sentences are "semantically equivalent" depends on context, perspective, and domain. Physical measurements are different. "5 kilograms" is either the same quantity as "5000 grams" or it is not. There is no ambiguity, no subjectivity, no context-dependence. This makes physical measurement systems what the authors call an "idealized testbed" for studying whether embedding spaces capture real-world structure.

The authors formalized the problem as follows. Given an embedding function that maps text to vectors, two quantities q1 and q2 measured in units u1 and u2, the cosine similarity is computed as the dot product of their embeddings divided by the product of their magnitudes. In a well-behaved embedding space, this similarity should correlate with the physical distance between the quantities after unit conversion. In practice, it does not.

The study tested four physical quantities: length (meters), mass (kilograms), volume (liters), and time (seconds). For each quantity, five numerical scales were used: Local (0 to 10), Medium (0 to 1000), Log (up to 100,000), Sign (-100 to 100, including negative numbers), and Scientific (exponential notation). The Local scale was expressed both numerically ("5 meters") and in words ("five meters") for integers from zero through ten, plus one hundred. Each range was broken into roughly 20 steps, producing value pairs like "2.5 meters" versus "8 meters" for which cosine similarity was computed.

The 24 Models Tested

The authors selected a diverse set spanning encoder-based and decoder-based architectures, older and newer releases, and multiple model families. The full list includes four Sentence-Transformers baselines (all-mpnet-base-v2, all-MiniLM-L6-v2, paraphrase-multilingual-mpnet-base-v2, LaBSE), ten contrastive BERT-style encoders (e5 variants, bge variants, mxbai, granite embeddings, nomic-embed), four ModernBERT-based encoders (DenseOn, granite-embedding-r2 variants), and six models derived from LLMs (Qwen3-Embedding at 0.6B/4B/8B, harrier-oss variants, embeddinggemma-300m). Embedding dimensionalities ranged from 384 (MiniLM) to 4096 (Qwen3-Embedding-8B). The authors frequently highlight all-mpnet-base-v2 (encoder) and Qwen3-Embedding-0.6B (decoder) as representative examples, but present parallel results for all 24 models in the appendix.

The Checker Pattern: Surface Similarity Wins

The most striking visual finding is what the authors call the "checker pattern." When plotting cosine similarity heatmaps for the Local scale (0 to 10), integer-integer pairs align well with each other while integer-float pairs do not, producing a checkerboard texture across the heatmap.

The practical consequence is counterintuitive. For both all-mpnet-base-v2 and Qwen3-Embedding-0.6B, the string "2.5 meters" receives a higher cosine similarity to "7.5 meters" than to "3 meters." Physically, 2.5 is closer to 3 than to 7.5 by a factor of five. But "2.5" and "7.5" share the character "5" and the token fragment ".5", while "2.5" and "3" share almost nothing at the surface level. The embedding models are treating measurement expressions as lexical strings rather than as quantities with inherent numerical meaning.

This pattern holds across all four physical quantities and does not improve with model size or recency. The authors note that older models considered "weaker" and newer LLM-based models considered "stronger" exhibit the same fundamental misalignment. The checker pattern is a direct visual signature of surface-form-driven similarity overpowering numerical semantics.

When Numbers Meet Words, Confusion Reigns

The authors also tested whether embedding models recognize that "one" and "1" denote the same quantity. In an ideal embedding space, these should map to nearly identical vectors. In practice, the alignment is weak and model-dependent.

all-mpnet-base-v2 assigns a markedly higher similarity to the pair "one" and "1" than to "one" and any other digit. This is sensible. But Qwen3-Embedding-0.6B does something stranger: it assigns high similarity to both "one/1" and "one/0", particularly for liter and kilogram quantities. It also shows pronounced horizontal lines in its heatmap when aligning digits with words. "Nine liters" is rated dissimilar to all other digit-expressed liter quantities, while "one liter" and "two liters" do not show this anomaly. Even the self-comparison of "nine" with "9" and "ten" with "10" is comparatively weak in Qwen, revealing that the model's internal representation of certain numbers is inconsistent with their mathematical identity.

Overall, embeddings barely capture the semantic equivalence between numerical and literal expressions of the same number. The word "one" and the digit "1" are treated as substantially different objects by most models, despite denoting the same quantity.

Unit Conversion: No Line Is Horizontal

Perhaps the most practical test of physical understanding is unit conversion. If an embedding model truly understands measurement, then "1 meter" should be nearly identical in embedding space to "100 centimeters," "1000 millimeters," and "0.001 kilometers." The authors plotted these conversion alignments as lines on a graph, where a perfect result would be a flat horizontal line at cosine similarity 1.0.

No model produced anything close to this. The lines are diffuse and erratic in both all-mpnet-base-v2 and Qwen3-Embedding-0.6B. In all-mpnet-base-v2, there is a strong drop-off in alignment as quantities grow larger. Qwen shows somewhat more consistent alignment, with the seconds-to-milliseconds conversion being the only line that exceeds 0.9 similarity and stays consistently high. But even this best case is not a flat line at 1.0, and all other conversion pairs show substantial noise.

The authors tested conversions across all four quantities and five scales, and the pattern is consistent: unit conversion is highly fuzzy in every embedding model. An embedding model cannot reliably tell you that 2 kilometers equals 2000 meters, which is precisely the kind of relationship that should be trivially captured if the model understood measurement semantics.

PhysScore: A Benchmark for Physical Understanding

To compare models on a single axis, the authors constructed a focused benchmark called PhysScore. For every pair of measurement expressions across all scales and quantities, they checked whether the cosine similarity correctly ranks the physically closer expression higher. Performance was measured using Kendall's tau (a non-parametric rank correlation) and Pairwise Accuracy (PAC), which counts the fraction of correctly ordered pairs.

The results are sobering. The best model is multilingual-e5-large-instruct with a Kendall's tau of 53.23 and a PAC of 76.61%. No model exceeds a tau of 54. For reference, a random baseline achieves a tau of 0 and a PAC of 50%. The average across all 24 models is a tau of 37.59 and a PAC of 68.79%.

The ranking does not follow model size or recency in any obvious way. nomic-embed-text-v1.5, a relatively small model, scores the worst at 18.57 tau. LaBSE, a model focused on cross-lingual similarity, scores 51.31, nearly matching the top performer. The Qwen family shows a counterintuitive pattern: the smallest Qwen3-Embedding-0.6B scores 40.47 tau, while the largest Qwen3-Embedding-8B scores only 32.27. Scale does not help.

Which Measurements Are Hardest?

The authors broke down performance by unit and range. Mass (kilograms) is the most difficult quantity for embedding models, with an average tau of only 34.54. Some models produce negative tau values for mass measurements, meaning they rank physical distances inversely from the truth. Distance (meters) and time (seconds) fare somewhat better, with average taus of 38.48 and 40.30 respectively.

Among numerical ranges, the medium scale (0 to 1000) proved most challenging. The log scale, which uses exponential spacing, produced the highest average performance for liters, meters, and seconds (targs around 61-62 tau), but not for kilograms (50.47). The maximum achievable tau for any single unit-range combination was 89.33 for seconds at the log scale, achieved by the best model. But this is the exception, not the rule.

Is the Similarity Function Just Miscalibrated?

A natural objection is that cosine similarity might not be the right metric. Perhaps the embedding vectors contain useful physical information, but it gets blurred when every dimension is weighted equally in the dot product. To test this, the authors trained a linear probe (regression model) on the absolute difference between embedding vectors and re-evaluated performance. This allows certain dimensions to be weighted more heavily than others.

The results in Table 3 show that recalibration helps only marginally. The average improvement across all models is just +3.0 tau. The largest increase is +12.3 for e5-large-v2, but its absolute score rises from 35.7 to only 48.1. Several models actually get worse with the probe (embeddinggemma-300m drops by 7.3, granite-embedding-107m-multilingual drops by 8.9). No model exceeds 54 tau even after recalibration.

The authors conclude that the misalignment is not caused by a poorly chosen similarity function. The information about physical measurement is simply not well-represented in the embedding vectors in the first place.

String Similarity Is the Real Driver

The authors' most revealing experiment investigates whether embedding similarity tracks string overlap or actual numerical proximity. They generated random numbers as floating-point and integer strings, then computed three similarity measures for every pair: character-level Levenshtein distance, token-level Levenshtein distance (using each model's own tokenizer), and ground-truth numerical distance. Kendall's tau was computed between embedding similarity and each of these three reference similarities.

The results in Table 4 show a clear pattern. Across nearly all models, embedding similarity correlates more strongly with string similarity than with numerical proximity. Consider embeddinggemma-300m: its character-level correlation (All) is 51.6, while its numerical correlation is 3.3. This model essentially treats "123" and "124" as more similar than "123" and "1000" because the former share more characters.

Qwen3-Embedding-0.6B shows the same pattern: character correlation of 47.6 versus numerical correlation of 11.7. Even the best-performing model on PhysScore, multilingual-e5-large-instruct, shows character correlation of 39.7 versus numerical correlation of 26.9. The string-overweighting is weaker there, but still present.

One outlier is nomic-embed-text-v1.5, which shows a negative character-level correlation of -5.6 and a negative token-level correlation of -16.9. This model appears to have learned some form of inverse string similarity for numbers, though it still fails to capture numerical proximity well (numeric correlation of 6.5).

What This Means for Developers Building with Embeddings

If your application involves any numerical or quantitative reasoning, these results should give you pause. Retrieval systems that embed product descriptions containing measurements (weights, dimensions, prices, durations) will rank results based on surface string similarity rather than actual quantity proximity. A search for "under 5 kg" could retrieve items described as "5.2 kg" (sharing the digit 5) over "4.8 kg" (physically closer but textually different). Recommendation engines, scientific literature search, and agentic systems that reason about quantities will all inherit this limitation.

The authors suggest that physical measurement accuracy could serve as an additional quality metric in future model development. For practitioners, the practical takeaway is to treat embedding similarity for quantitative expressions with skepticism. If accurate measurement semantics matter for your use case, you may need to build explicit numerical parsing into your pipeline rather than relying on embedding similarity alone.

The underlying issue, the authors argue, is the contrastive training objective itself. It provides insufficient pressure to organize the embedding space according to physical or quantitative relationships. For such alignment to emerge, it would either need to be an emergent capability (which this study shows it is not, across 24 models) or an explicit training objective. The observation that lexical similarity is a useful inductive bias for identifiers, version numbers, and product codes suggests that the current behavior is not a bug from the training perspective. But it is a limitation for any system that expects embeddings to capture the meaning behind numbers.

As embedding models are increasingly deployed in scientific, engineering, and agentic systems that require reasoning about quantities, this gap between surface similarity and semantic understanding of measurement becomes more consequential. The ability to correctly judge that "2 kilometers" is semantically equivalent to "2000 meters" is elementary, yet no model tested comes close to achieving it reliably.

Read the paper on arXiv