When an LLM Cites a Source, Can You Actually Verify It?
Clinicians using LLMs for question answering face a practical problem. The model produces an answer with citations, but the citations point to broad texts, entire guidelines, or full sections. A busy clinician cannot verify a specific claim by opening a 200-page guideline and scanning for the relevant passage. The alternative is verifiable-by-construction design: every factual claim in the answer carries a verbatim quote from the source material, so the reader can verify the claim without opening other documents. Zhang, Chen, Commodore-Mensah, and Oberst from Johns Hopkins evaluate twelve LLMs on this end-to-end task: from providing citations for every factual claim, to producing verbatim quotes, to ensuring those quotes fully substantiate the claims. The results are sobering. Most models can attach quotes to over 90% of claims, but the quotes often fail to support every detail of the claims they accompany.
The Claim Funnel: Four Stages of Verifiability
The evaluation framework decomposes verifiability into four sequential stages, each passing the output through a filter that removes unsupported claims. The first stage is claim identification: given an answer with inline citation markers, strip the markers, split the prose into sentences, and classify which sentences contain checkable clinical claims. The second stage is citation coverage: does each claim sentence have a citation attached, using an attribution window of one sentence (each citation applies to the immediately preceding claim). The third stage is verbatim compliance: does the quoted text actually appear in the cited source section, either as an exact character-for-character match, or under relaxed conditions allowing case normalization and ellipsis? The fourth stage is claim support: given the claim and its accompanying verbatim quotes (and nothing else), does the quote fully substantiate every detail of the claim?
The certified claim rate (CCR) is the share of all claims that pass all four stages: they carry at least one verbatim-compliant quote and are fully supported by that quote. This is the metric that matters for a clinician trying to verify an answer without opening external documents.
The Setup: Four Guidelines, 222 Questions, Twelve Models
The corpus consists of four clinical practice guidelines: the 2019 AHA guideline on primary prevention of cardiovascular disease, the 2025 AHA guideline on high blood pressure, the 2026 AHA guideline on blood cholesterol, and the 2026 ADA Standards of Care in Diabetes. Together these cover the principal modifiable risk factors for atherosclerotic cardiovascular disease. From these guidelines, 222 synthetic clinical questions are generated using GPT-4o-mini, with each question derived from a particular subsection of a particular guideline. This design ensures that questions are answerable from the provided reference material and that retrieval quality can be verified against ground-truth sections.
The clinical QA system has two components. An LLM-guided semantic retrieval stage uses an LLM to select relevant sections based on section summaries, substantially outperforming both dense-vector and lexical baselines in recall of the ground-truth section. The retrieval output is frozen: deepseek-v4-flash performs a one-time retrieval, and the shared context is the same for all generation models. This eliminates retrieval quality as a confound when comparing generation models. The generation stage receives the query and retrieved context, with a system prompt instructing the model to answer in short prose and attach an inline citation marker to each clinical claim. The marker pairs a section identifier with a verbatim quote from that section, written as a structured template that a UI could render as clickable citation indicators.
Twelve LLMs from five vendors are evaluated: GPT-5.4, GPT-4.1, Claude Opus 5, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 2.5 Pro, Gemini 2.5 Flash, DeepSeek V4, DeepSeek V4 Flash, Qwen3 235B, Llama 4 Maverick, and Mistral Large. Each model generates answers to the same 222 questions under identical retrieval conditions.
Results: Quotes Are Easy, Substantiation Is Hard
The claim funnel reveals where models succeed and where they fail. Most models achieve citation coverage above 90%, meaning nearly every claim sentence has a citation attached. The prompt instructs the model to cite every claim, and most models comply. Lightweight models like Claude Haiku 4.5 are exceptions, producing fewer citations per claim.
Verbatim compliance is the next filter. Most models produce quotes that pass the verbatim check at rates above 85%. The quotes actually appear in the cited source sections. But the details matter. Claude Opus 5 produces verbatim quotes for 98.0% of its claims, one of the highest rates. GPT-5.4 achieves around 93%. The verbatim check is deterministic: it checks whether the quoted text appears in the cited section, allowing for case normalization and minor formatting differences.
The critical bottleneck is claim support. Given the verbatim quote and the claim (and nothing else), an LLM judge decides whether the quote fully substantiates every detail of the claim, partially substantiates it, does not support it, or contradicts it. Claude Opus 5, despite its 98.0% verbatim compliance, fully substantiates only 37.1% of its claims. The quotes are real text from the source, but they do not contain enough information to support every point the claim makes. GPT-5.4 performs best overall, with a certified claim rate around 75%, meaning three-quarters of its claims carry a verbatim quote that fully supports them.
The decomposition reveals different failure modes. Claude Haiku 4.5 achieves a certified claim rate of only 24.9%, driven largely by failures to produce faithful verbatim quotations. It quotes text that does not appear in the source. Claude Opus 5, by contrast, produces faithful quotes but fails to provide enough context. The quote is real but incomplete. This is a fundamentally different problem: the model knows what the source says but does not quote enough of it to substantiate the full claim.
Why This Gap Matters for Clinical Use
The gap between citation coverage and certified claim rate is the core finding. A clinician looking at an answer from Claude Opus 5 would see citations attached to 98% of claims, with verbatim quotes in those citations. The quotes are real text from the cited guidelines. But only 37.1% of those claims are fully substantiated by the quotes. The remaining 62.9% require the clinician to open the guideline and find additional supporting text. The citations create a false sense of validation: the presence of verbatim quotes suggests the answer is fully grounded, but the quotes often omit details the claim asserts.
This is the verification problem the paper addresses. Current citation systems point to broad texts, requiring the reader to locate the evidence. Verbatim citation is a finer granularity: the quote is right there, next to the claim. But if the quote does not fully substantiate the claim, the reader still has to open the source. The paper shows that even when models produce faithful verbatim quotes, the quotes frequently lack the specificity to support the full claim.
Model-Specific Patterns
The framework's decomposition allows comparing models not just on overall accuracy but on where they fail. The strongest systems, GPT-5.4 and Gemini 2.5 Pro, balance citation coverage, verbatim compliance, and claim support relatively well, achieving certified claim rates around 70-75%. They attach citations to nearly all claims, produce faithful quotes, and those quotes tend to substantiate the full claim.
Claude Opus 5 shows the most striking gap between verbatim compliance (98.0%) and claim support (37.1%). It is excellent at reproducing source text but poor at selecting quotes that cover the full scope of its claims. This suggests the model generates claims that are broader than any single quote in the source, or selects quotes that are too narrow to support the claim's assertions.
Claude Haiku 4.5 fails earlier in the funnel. It does not produce faithful verbatim quotes, with a verbatim compliance rate well below the other models. The lightweight model lacks the capability to reproduce source text accurately, which is a prerequisite for verifiability.
The framework also tests sensitivity to the attribution window. Using k=1 (each citation applies only to the nearest preceding claim) is the strictest setting. Larger windows allow a citation to cover multiple claims, which could increase citation coverage but also obscure which claim each quote supports. The main results use k=1 to maintain the one-to-one correspondence between claims and quotes that a clinician would expect.
Implications for Building Verifiable Clinical QA
The paper suggests that current LLMs are not yet capable of producing answers that are verifiable by construction in clinical settings. The best systems support about three-quarters of claims with fully substantiating verbatim quotes. The remaining quarter requires the reader to open external documents. For a clinician making decisions based on these answers, the gap between 75% and 100% is the difference between trusting the answer and having to verify it manually.
The decomposition also points to specific areas for improvement. For models like Claude Opus 5, the bottleneck is quote selection: the model produces faithful quotes but selects ones that are too narrow. For models like Claude Haiku 4.5, the bottleneck is quote generation: the model cannot reproduce source text accurately. These are different technical problems requiring different solutions.
The artifacts released with the paper, including the synthetic question set and evaluation harness, provide a standardized way to measure progress on this specific capability. The framework can be applied to new models as they are released, and the clinical guideline corpus provides a realistic testbed for verifiability in a high-stakes domain.
The broader lesson is that citation alone is not sufficient for verifiability. The granularity of the citation matters, and even at the verbatim-quote granularity, the quote must substantiate the full claim. Current models fall short on both dimensions in different ways, and the gap between citation coverage and certified claim rate is the metric that matters for practical deployment.
Read the paper on arXiv