When an LLM reports a molecular property with a median absolute error of 0.025 kcal/mol on the FreeSolv benchmark, that number is far below the 0.6 kcal/mol experimental uncertainty assigned to the measurements. No model can predict a property to that precision from molecular structure alone. What it is doing is reproducing the published number from memory. This paper systematically audits 22 frontier language models across 12 molecular property regression benchmarks and finds that verbatim retrieval of published values is widespread, benchmark-specific, and amplified by reasoning.
Why Accuracy on Molecular Benchmarks Is Not What It Seems
LLMs are increasingly evaluated on molecular property prediction tasks: estimating solubility, toxicity, boiling points, and other quantities from SMILES strings. When a model achieves low error on these benchmarks, the standard interpretation is that it has learned something about molecular physics. But accuracy on a regression benchmark cannot distinguish a model that predicts a property from one that retrieves a memorized number. If the published value of a molecule is recoverable from the model's weights, a low error measures exposure to the literature rather than generalization from chemical principles.
The distinction matters because the third significant figure of an experimentally measured value is dominated by measurement noise and is not predictable from molecular structure. A model that reproduces the third digit more often than chance has memorized the value, not learned the physics. The authors build their detection framework on this insight.
A Digit-Level Detection Framework
The method works by comparing predictions against a molecule-blind null hypothesis. For each molecule, the prediction and the published value are compared at 1, 2, and 3 significant figures. This produces nested counts: how many predictions match at one figure, how many of those survive to two figures (retention R12), and how many of those survive to three (retention R23). The null hypothesis is the best a molecule-blind procedure can achieve: a predictor that knows only the digit distribution of the labels, sees no structure or formula, and does best by emitting the modal continuation of whichever prefix it has already matched. This floor is fitted on a random half of the labels and scored on the other half, so fitting and evaluation do not share data.
A retention significantly above this floor means the model resolves digits no molecule-blind procedure can resolve. In principle an accurate predictor could reach the same retentions without retrieval, but the third significant figure of experimental data is not predictable from structure. The models' own predictions support this: on the four retrieved benchmarks, their correlation with published values does not reach that of a reference predictor that gets exactly the first significant figure right and knows nothing below it, in 87 of 88 cells. The general predictive capability of the models is not sufficient to explain agreement on the second and third figures.
Each cell is tested against the null using a binomial test, with Benjamini-Hochberg correction for multiple testing across all cells in a run. The two retentions of a cell are computed from nested counts on the same molecules and are positively dependent, so the false discovery rate is controlled under positive regression dependence.
The Contamination Map
The main experiment covers 22 models (including Claude Opus 4.6, Sonnet 5, Haiku 4.5; GPT-5.4, 5.5, 5.6 in luna/terra/sol variants; Gemini 3 Flash, 3.1 Flash-Lite, 3.5 Flash; and others) across 12 benchmarks (FreeSolv, ESOL, LD50, AqSolDB, BACE, Caco-2, QM7, QM9, boiling points, and others). Each model is queried zero-shot on the same 500 molecules per benchmark at a controlled reasoning level of approximately 1,024 tokens. The prompt names the benchmark and asks for a single number to three significant figures.
The results show that retrieval is concentrated on a few datasets. FreeSolv, ESOL, LD50, AqSolDB, and the boiling points show significant retrieval across many models. The remaining datasets show only isolated flagged cells. On five datasets, more than 50% of models show verbatim retrieval. On the others, it appears in scattered cells.
The boiling points are the clearest case: they are available in full on Wikipedia, and the model reproduces nearly all of them. A sweep over three open pretraining indices supports this reading: how often a benchmark's molecules appear across documents tracks its strongest retrieval rate (rho=0.88 over the twelve benchmarks), while the number of documents carrying the benchmark file's own column headers does not (rho=0.20). The retrieval likely comes from secondary sources that reproduce parts of the datasets rather than the complete benchmark files.
Model size shows no apparent influence. In-family comparisons, such as the three GPT-5.6 variants, yield similar contamination levels. Older models like GPT-4.1 and Gemini 2.5 Pro are less contaminated than current frontier models. Why contamination increased with the most recent models remains unclear.
Reasoning Amplifies Retrieval
The same experiments, on the same molecules with the same prompt, are flagged 89% more often at a higher reasoning level than at the lowest one. At minimum reasoning, 47 of 264 cells show significant retrieval. At the standard reasoning level, 89 cells do. The increase is concentrated on the retrieved benchmarks: LD50 goes from 4 flagged cells to 16, ESOL and AqSolDB from 4 to 15, while the recency control, BACE, and Caco-2 stay at 0.
A five-model reasoning ladder (Gemini 3 Flash, Gemini 3.1 Flash-Lite, Gemini 3.5 Flash, GPT-5.5, Claude Opus 5) at up to five reasoning settings each confirms this. Retrieval increases with reasoning tokens for all datasets where significant retrieval is found. For most models and datasets, the rate starts near zero at zero reasoning tokens and rises as more tokens are emitted. Datasets without retrieval show flat rates regardless of reasoning level.
The mechanism is not fully established. On LD50, the published value is a unit conversion away from the measured one, involving molecular weight calculation and a logarithm. Reasoning traces captured at high effort show this conversion occurring, suggesting that reasoning supplies the arithmetic needed to recover the stored value. On benchmarks without unit conversion, the increase is unexplained.
Claude Opus 5 is distinct: it achieves the highest retrieval rates on FreeSolv and ESOL (hit3 approximately 65%) while emitting one to two orders of fewer reasoning tokens than other models. All models emit fewer reasoning tokens on datasets they know best, while reasoning still helps restore values stored in the weights.
This has a practical consequence for contamination audits. An audit at minimum reasoning misses about half of contaminated cases. Every contamination map is a lower bound at the setting it was run at.
Can Retrieval Be Interrupted?
The authors test three interventions. Rewriting each molecule to a different valid SMILES string for the same structure leaves a median 102% of retrieval in place and clears none of twelve tested cells. Retrieval is keyed to the molecule rather than to the exact string the benchmark publishes. Scoring at a low reasoning level lowers the measured rate without removing retrieval.
The third intervention is a blinding protocol that transforms the characters of the SMILES string and names the target only as "the sample property" in the prompt. This is tested in an in-context learning setting with 100 labeled examples on four models and three benchmarks. Blinding reduces verbatim retrieval in every cell but one. Ten of twelve cells are flagged under the unblinded condition and three under blinding. In nine of twelve cases, hit3 falls below 2%.
The exception is GPT-5.6 sol on ESOL, which moves from 14% to 13% and is essentially untouched. Claude Opus 5 is reduced but not cleared: from 48% of ESOL and 31% of FreeSolv under the original SMILES, it still reproduces 6.7% and 4.8% from a structure string it has never seen, and both cells stay flagged. The strongest models in some cases still recognize a combination of transformed SMILES strings and original labels, probably recognizing the dataset from the labels of the in-context examples rather than from the structures.
What Happens to Benchmark Scores When Retrieval Is Suppressed
The blinding experiment also measures how suppression changes benchmark scores. Suppressing retrieval moves the prediction errors of different models closer together in relative terms, while their differing use of verbatim retrieval spreads them apart. This indicates that the general predictive capability of an LLM is not determined solely by the amount of memorized values. Some models that score well on molecular benchmarks do so partly through retrieval; others that score lower may have comparable or better predictive capability once retrieval is accounted for.
The practical implication is that benchmark rankings for molecular property prediction are not reliable indicators of chemical reasoning ability. A model with high retrieval rates may score well while having limited ability to predict properties of molecules it has not memorized. A model with lower scores but less retrieval may generalize better to novel compounds.
Limitations and Scope
The study is limited to 22 models, 12 benchmarks, and zero-shot evaluation with a specific prompt format. Different prompts, few-shot settings, or fine-tuned models may show different patterns. The detection framework relies on the assumption that the third significant figure of experimental data is not predictable from molecular structure, which holds for measured properties but not necessarily for computed ones. The recency control (antiviral potencies from a 2024-2025 blind challenge) postdates the knowledge cutoff of three models, but for most models the cutoff is later or unstated.
The paper does not assess whether retrieval is learned from benchmark files specifically or from secondary sources that reproduce parts of the data. The corpus prevalence analysis suggests secondary sources are the more likely route, since document-level prevalence of benchmark molecules tracks retrieval while prevalence of benchmark column headers does not.
For practitioners evaluating LLMs on molecular tasks, the core recommendation is that accuracy alone is insufficient. A contamination check using digit-level retention statistics should accompany any molecular property benchmark result. The authors provide the framework and show that it can be run at minimum reasoning as a cheap lower bound, though it will miss roughly half of contaminated cases.