Evaluating how well large language models retrieve information from electronic health records has been a persistent challenge. Existing benchmarks are manually curated, expensive to build, and rapidly become obsolete as clinical documentation practices evolve and models absorb benchmark content into their training data. A team of researchers from Stanford University — led by Jordan Cahoon, Chloe Stanwyck, and Emily Alsentzer, alongside collaborators from Stanford Medicine and the broader biomedical data science community — introduces BRIE, the Benchmark for Retrieving Information in EHRs, a "living" benchmark that can be automatically regenerated from new patient data with minimal human intervention.
The Evaluation Problem in Clinical AI
Clinicians are increasingly interacting with LLM-based assistants embedded directly within electronic health record systems. These tools can answer questions about a patient's medical history, locate records from prior encounters, and generate summaries. But evaluating whether these systems actually work — and where they fail — has lagged behind deployment. Traditional benchmarks rely on clinicians manually authoring question-answer pairs, a process that is time-consuming, expensive, and scales poorly. Once created, static benchmarks degrade: as models train on them, they begin to reflect memorization rather than genuine retrieval ability. As documentation practices shift and new types of clinical encounters emerge, the benchmarks lose their representativeness.
The specific challenge of clinical information retrieval is distinctive. Patient records are long, noisy, and span many encounters. Relevant information is frequently duplicated through copy-forward documentation, where notes import text from previous entries. Clinical observations may be interleaved with findings from unrelated visits, making it difficult to determine when something was documented. The relevant fact may be buried hundreds of thousands of tokens back in a patient's history. Extracting signal from this noise is fundamentally harder than answering questions from a single focused document.
How BRIE Is Built
BRIE takes a different approach. Rather than asking clinicians to write questions from scratch, the team built a scalable LLM-based generator that automatically produces question-answer pairs from de-identified longitudinal clinical notes. The generator operates in two stages. First, it extracts and deduplicates candidate facts from all clinical notes documented before a patient's admission History & Physical (H&P) note, removing redundancy introduced by copied or imported text. Then, these facts and the H&P note are jointly provided to an LLM in a single prompt to generate question-answer pairs, where answers are directly linked to a subset of patient facts.
The design ensures that every answer must be supported by the extracted facts from the prior record rather than by information in the H&P itself. This means the questions genuinely test retrieval from the longitudinal history, not simply comprehension of the encounter summary. Each question is generated along an evaluation taxonomy with three axes: reasoning complexity (single-hop versus multi-hop), clinical topic (such as comorbidities, diagnostic testing, or radiology), and temporality (how far back in the record the supporting evidence resides).
The critical question is whether automatically generated questions are clinically useful. To answer this, the team recruited fifteen physicians with two independent reviewers per question. Reviewers evaluated each generated entry across fourteen criteria for clinical relevance and accuracy. The results were encouraging: 83.7% of questions were judged clinically relevant, 97.8% were consistent with the patient chart, and 92% of generated answers were judged accurate and complete. Natural phrasing was the primary area for improvement, at 73.3%, with reviewers supplying alternative phrasings where needed.
After a third physician adjudicated disagreements about clinical relevance, the final BRIE dataset contained 508 high-confidence questions drawn from 675 generated pairs, spanning 63,878 de-identified clinical notes from 68 medically complex patients at Stanford Health Care.
What Models Get Wrong
The team evaluated nine LLMs — including Claude Opus 4.7, Gemini 2.5 Pro, GPT 5.4, and several open-weight models — under five inference configurations: recent-context baselines (Recent, Recent-180K), retrieval-augmented generation methods (BM25 sparse retrieval, Dense semantic retrieval), and an agentic configuration with iterative tool use. The models were assessed using an automated LLM-as-a-judge framework based on fact entailment, measuring fact recall (what fraction of reference facts the model surfaces) and fact precision (what fraction of the model's claimed facts are supported by the record).
The headline finding is that fact recall was moderate across all configurations, ranging from 0.28 to 0.78. Even the strongest model-inference combination left roughly a quarter of clinician-verified supporting facts unsurfaced. But the more revealing finding is that the dominant failure mode is not hallucination — it is omission. In a random sample of 50 queries spanning models and configurations, 99.2% of extracted model response facts were faithful to the clinical record. Models were verbose but not fabricated. They simply left things out.
These omissions are particularly dangerous in a clinical context. A response may appear plausible and accurate while silently excluding critical information that would only become apparent through careful review of the entire patient chart. As the authors note, omission errors are difficult to detect and may be more consequential than fabrication.
Where Retrieval Breaks Down
By decomposing performance across the BRIE taxonomy, the researchers revealed that failures are concentrated rather than uniform.
Multi-hop reasoning is the hardest. Questions requiring synthesis across multiple encounters consistently yielded lower fact recall than single-hop questions. The average decrease in fact recall for multi-hop questions was –0.20 for agentic retrieval, –0.14 for BM25, and –0.13 for Dense retrieval. These are questions like asking about prior rashes and treatments in a transplant patient, where the answer requires combining information from different notes spanning weeks or months. Models routinely omitted clinically relevant details about treatment effectiveness or dose-dependent relationships.
Longitudinal evidence is hardest to find. For 63 questions, the earliest supporting evidence occurred more than 180,000 tokens before the query time — beyond the context window of most long-context models. When the earliest supporting fact appeared beyond this cutoff, fact recall fell by an average of 0.16 and 0.25 for Recent and Recent-180K inference respectively, but no significant drop was observed for Dense or BM25 retrieval, which can actively search the record.
Clinical topic matters. Recall was lower for questions about comorbidities (0.68) and disease progression (0.67) than for imaging (0.76) and diagnostic testing (0.76). Comorbidities and disease progression may be documented across multiple encounters with varying levels of detail, making it difficult for models to distinguish separate events and construct an accurate timeline.
Dense Retrieval Matches Long Context at Lower Cost
One of the most practically significant findings is that Dense retrieval achieved performance comparable to long-context inference while processing an average of 329,697 fewer tokens — a 69% reduction in context length. For cost-efficient models, the gains were substantial: Dense retrieval raised average fact recall by 0.18 for Gemini 2.5 Flash Lite, 0.06 for Claude Haiku 4.5, and 0.03 for GPT 5.4 Nano, relative to recent-context inference.
The paper provides several illustrative cases where Dense retrieval made the difference. For a patient presenting with cough and dyspnea, Dense retrieval surfaced a 2019 pulmonary function test documenting severe airflow obstruction that Recent inference missed entirely. When asked for the most recent bone marrow biopsy in a patient with myelofibrosis, Dense retrieval accurately recovered the most recent report showing normal bone marrow function, while Recent inference returned contradictory claims about whether the biopsy existed. In each case, the relevant evidence was present in the record but difficult to identify because of duplicated text and temporal ambiguity.
Agentic retrieval, despite being a promising approach, did not improve upon Dense retrieval. For Claude models, agentic retrieval even underperformed the simpler Recent inference baseline, reducing fact recall by 0.08 and 0.04 respectively.
Multiple Answers Reveal Hidden Capability
A particularly innovative aspect of BRIE is its support for multi-reference evaluation. The researchers generated alternative interpretations for 100 sampled questions, with four board-certified physicians evaluating the results. Eighty-seven questions yielded at least one additional valid interpretation, with an average of 3.18 valid interpretations per question. When model responses were scored against the most favorable interpretation rather than a single reference answer, fact recall increased by 0.12 — a relative improvement of 31.4%. Fact precision increased by 0.14, a relative improvement of 45.4%.
This matters because clinical questions often have multiple valid answers depending on context or clinician perspective. A response that omits one clinically relevant detail may still be a valid answer to a differently framed question. Single-reference evaluation systematically underestimates model capability by penalizing responses that are aligned with alternative valid interpretations. However, even when adjusting for this, state-of-the-art systems still exhibited omission errors.
A Living Benchmark That Stays Current
To demonstrate that BRIE can be continuously updated without clinician intervention, the team regenerated the benchmark from admissions collected two years later, producing BRIE-new with 1,000 questions. Performance was stable across all three cohorts — BRIE, BRIE-unfiltered, and BRIE-new — with mean fact recall ranging from 0.70 to 0.73. More importantly, the characteristic failure modes persisted: reduced performance on multi-hop questions, questions requiring longitudinal reasoning, and temporally distant evidence retrieval remained consistent across all cohorts. The regenerated benchmark retained its difficulty without any human filtering.
This temporal robustness is crucial. A benchmark that degrades as models absorb its content or as documentation practices change is of limited practical value. BRIE's generator can be rerun on newly collected clinical records from any institution, producing updated evaluation datasets that reflect current clinical practice.
Limitations and the Path Forward
The authors acknowledge several limitations. BRIE is derived from a single institution's records, and cohort generalizability across institutions with different documentation practices and patient populations has not yet been established. The benchmark is derived from unstructured clinical notes; incorporating structured EHR data such as laboratory results, medications, and vital signs could enhance realism. The evaluation covers five inference configurations, and richer retrieval tools and reasoning strategies remain to be explored. Finally, the fact-based evaluation framework measures information coverage without weighting facts by clinical importance — a critical direction for future work.
The broader significance of BRIE is that it reframes clinical AI evaluation as an ongoing process rather than a fixed artifact. By making it feasible to continuously generate and validate benchmarks, it provides a foundation for identifying when and why clinical retrieval systems fail as models, data, and deployment settings evolve. In a field where silent omission of clinically relevant information can have direct consequences for patient care, the ability to rigorously and continuously evaluate retrieval systems is not merely an academic concern — it is a prerequisite for safe deployment.