When you ask an AI model to generate a chest X-ray report, how do you know if it did a good job? The standard answer: compare it to a reference report written by a human radiologist. If the machine output looks similar to the human reference, the model gets a high score. But this seemingly straightforward evaluation scheme has a hidden flaw that can flip model rankings entirely, and it has nothing to do with clinical accuracy.
The problem is that radiologists do not agree on how to write reports. Two radiologists examining the same chest X-ray, finding the same abnormalities, can produce reports that look nothing like each other. One might write a detailed narrative enumerating every normal structure: "The lungs are clear. No pleural effusion. No pneumothorax. The cardiac silhouette is normal in size." The other might write a single sentence: "No acute cardiothoracic abnormality." Both communicate the same clinical meaning. But when an AI evaluation metric compares a model-generated report against these two references, the scores can diverge sharply, because the metric measures not just what the report says, but how it says it.
A team from Carnegie Mellon University, Johns Hopkins, and Highmark Health set out to quantify this problem. Their paper, "Reporting Practice Matters," demonstrates that the choice of reference report can alter the rankings of nine radiology report generation (RRG) models on four datasets, using metrics that the field currently considers reliable. The effect is not subtle. On MIMIC-CXR evaluated with RadCliQ-v1, a metric designed to correlate with radiologist judgment, condensing the reference reports to omit normal findings causes Libra to drop from first place to second, while CheXOne rises from third to first. The underlying clinical findings have not changed. Only the reporting practice of the reference has.
Why evaluation metrics conflate style with substance
Most RRG evaluation metrics work by comparing a model-generated report to a single reference report. GREEN, CRIMSON, RadCliQ-v1, RadGraph-F1, and RaTEScore all fall into this category. Some use LLM judges (GREEN and CRIMSON), others extract clinical entities and measure overlap (RadGraph-F1), and others combine multiple sub-metrics into a composite score (RadCliQ-v1). The underlying assumption is that closer alignment with the reference indicates better clinical accuracy.
This assumption breaks down when two references describe identical clinical findings in different ways. GREEN, for example, matches individual findings between the generated and reference reports. When the reference enumerates normal findings ("No pleural effusion", "No pneumothorax"), GREEN checks whether the generated report mentions each one. If the reference simply states "No acute abnormality," GREEN has fewer individual findings to match, and the scoring changes. A model that happens to enumerate normal findings will score higher against the enumerated reference, even if its clinical accuracy is identical against both. The metric rewards stylistic conformity, not diagnostic correctness.
This is not a minor edge case. Radiologists' reporting practices vary along multiple dimensions simultaneously. The way a report is organized, how exhaustively it lists findings, how specific it gets about anatomy and measurements, and the terminology and hedging language it uses all vary across radiologists, institutions, and training backgrounds. Prior work by Weiss and Langlotz in 2008 and Bosmans et al. in 2014 documented this heterogeneity, but it has not been systematically accounted for in RRG evaluation until now.
A radiologist-informed taxonomy of reporting variation
The authors constructed a taxonomy of reporting practice variations under the guidance of a board-certified radiologist. The taxonomy has four top-level dimensions, each containing specific axes along which reports commonly differ.
Organization covers how report content is arranged. The Structure axis contrasts templated formats (headings, bullet points, subheadings) with free-text prose. The Section Assignment axis captures where radiologists place content. Some put differential diagnoses in the Findings section, others in the Impression, others split it between both.
Completeness captures how exhaustively a report describes findings. The Findings Scope axis contrasts an exhaustive style that lists every normal and abnormal finding against a minimal style that focuses only on clinically relevant abnormalities. This is the axis responsible for the "No acute cardiothoracic abnormality" versus enumerated normals example.
Granularity covers the level of detail. Anatomical Granularity contrasts reports that specify "pulmonary nodule in the right upper lobe anterior segment" with those that aggregate to "right lung pulmonary nodule." Quantitative Granularity contrasts explicit measurements ("cardiothoracic ratio of 0.55") with qualitative descriptions ("mild cardiomegaly").
Language captures linguistic variation. Terminology covers near-synonymous phrasings like "cardiomegaly" versus "enlarged cardiac silhouette." Hedging covers different expressions of diagnostic uncertainty: "worrisome for" versus "concerning for." Negation contrasts explicit negation ("no pneumothorax") with implicit positive phrasing ("pleura appear normal").
Each axis represents a dimension along which a radiologist could write a valid, clinically equivalent report using a different reporting style. The taxonomy is not exhaustive, but it captures the axes that most commonly produce superficially different reports from the same underlying interpretation.
ReRef: rewriting references while preserving clinical meaning
To study how these reporting practice variations impact evaluation, the authors needed a way to generate alternative reference reports that change reporting style while keeping clinical content identical. They built ReRef, a two-stage LLM-based pipeline that takes a reference report and a target axis (e.g., Findings Scope (Minimal)) and produces a rewritten alternative.
ReRef has two components: a Generator and a Verifier. The Generator, instantiated with OpenAI's o3 model, drafts an alternative reference report that applies the target variation. The Verifier, also o3, then checks two things. First, does the generated report actually simulate the target variation? Second, is it clinically equivalent to the original? Clinical equivalence means the overall diagnosis and implied patient management remain unchanged. If either check fails, or if the target variation is infeasible without hallucinating new information (for example, adding numerical measurements that were not in the original), the process retries up to five times.
The reason a Verifier is necessary is obvious once you consider the edge cases. Ask an LLM to rewrite a report with "Quantitative Granularity (High)" and it might invent a measurement that was never in the original report. Ask it to change the "Findings Scope" and it might accidentally drop an abnormal finding. The generation-verification loop prevents these failures.
The authors validated ReRef with 13 board-certified radiologists across 480 original-alternative report pairs drawn from four datasets. The results are strong. For realism (Task 1), radiologists rated the alternative reports as realistic or better in 92.7% of cases. For clinical equivalence (Task 2), 98.8% of alternative reports were rated as equivalent or having only minor discrepancies that would not change clinical management. Only 1.2% received a major discrepancy rating. The Minor Discrepancy ratings often involved changes in anatomical detail level that radiologists explicitly noted would not affect treatment decisions.
Model rankings shift under reporting practice perturbations
With ReRef validated, the authors generated alternative references for all four datasets along each axis of the taxonomy, then re-evaluated nine open-source RRG models. The models spanned diverse architectures and sizes: MAIRA-2, MedGemma-4B, CheXOne, MedVersa, CheXagent-2-3B, Med-CXRGen, Libra, LLaVA-Rad, and CheX-MIMIC. The evaluation covered five metrics the field considers correlated with radiologist judgment (CRIMSON, GREEN, RadCliQ-v1, RaTEScore, RadGraph-F1) plus four additional metrics (CheXbert-F1, BLEU-2, ROUGE-L, BERTScore).
The headline result: most metrics are sensitive to reporting practice variations in the reference, and the sensitivity is large enough to alter model rankings. The authors quantified this by computing the change in average model score for each axis-aligned perturbation, normalized by the metric's standard deviation across models and datasets. They found that the distribution of score changes, aggregated across all (model, dataset) pairs, spans a range that would be considered meaningful in any practical comparison.
The RadCliQ-v1 example is the most striking. When the Findings Scope axis is used to condense references to focus on key abnormalities, Libra's ranking drops from first to second place on MIMIC-CXR, while CheXOne rises from third to first. Neither model changed. Only the reference reports changed. This means that a leaderboard ranking for these models, as published, depends on which radiologist happened to write the reference report for each study.
The paper includes a concrete example on the IU X-ray dataset where a report generated by CheXagent is evaluated using GREEN against two alternative references from a Findings Scope (Minimal) perturbation. Under the original reference, which enumerates normal pleural findings, GREEN matches several findings and assigns a positive score. Under the alternative reference, which omits normal findings and states "No acute cardiothoracic abnormality," GREEN finds a mismatch in the granularity of pleural findings and assigns a score of zero. The model's output has not changed. The clinical content it describes is the same. But the metric penalizes it for not matching a reporting style that was not in the original reference.
Which metrics are most and least affected
Not all metrics respond equally to reporting practice variations. The authors report the mean and maximum interquartile range (IQR) of normalized score changes across all axes for each metric. NLG metrics like BLEU-2, ROUGE-L, and BERTScore show high sensitivity, because they directly measure lexical overlap, which is exactly what reporting practice variations alter. When a reference switches from enumerated normals to a single sentence, BLEU-2 and ROUGE-L drop substantially, regardless of clinical accuracy.
Among the metrics designed for clinical accuracy, RadCliQ-v1 and GREEN show moderate to high sensitivity. CRIMSON shows somewhat lower sensitivity across axes, though it is not immune. RaTEScore, which is based on clinical entity extraction, also shows meaningful variation. No metric fully decouples clinical interpretation from conformity to reporting practices.
The granularity axes (Anatomical and Quantitative) produce particularly interesting effects. When a reference report includes detailed anatomical locations and the alternative aggregates to the organ level, metrics that extract and match clinical entities can penalize a model for not being specific enough, even though the alternative reference was less specific. The model is being evaluated against a standard that varies not because the right answer changed, but because the reference's level of detail changed.
What this means for practice
The immediate implication is that reporting practice should be treated as a variable, not a constant, in RRG evaluation. When comparing models on a benchmark like ReXrank, the choice of reference reports matters as much as the choice of metric. A model that ranks first against references from one radiologist might rank third against references from another, not because it is worse, but because it was trained on or optimized for a different reporting style.
For researchers building or evaluating RRG models, the paper suggests several practical steps. First, when reporting benchmark results, specify the reporting practices of the reference reports, or at least acknowledge the sensitivity of results to reference choice. Second, when designing evaluation protocols, consider using multiple reference reports per study, or at minimum, ensure the references match the intended deployment setting's reporting conventions. Third, consider metrics that are more robust to reporting practice variations, or develop new metrics that explicitly decouple clinical accuracy from stylistic conformity.
The authors also release MIMIC-CXR-Ext-ReRef, a radiologist-validated dataset of 120 original-alternative report pairs from MIMIC-CXR, and open-source code for ReRef and all evaluations. This is directly useful for anyone working on RRG evaluation methodology.
On a longer timeline, this work points to a fundamental limitation of single-reference evaluation for free-form text generation tasks. When the space of valid outputs is large and includes stylistic variation, comparing against a single reference conflates "did the model get the right answer" with "did the model write in the same style as this particular person." For radiology report generation, where clinical accuracy is what matters, the field needs evaluation methods that can separate these two concerns. ReRef and the taxonomy of reporting practice variations provide a concrete framework for doing so.
Read the paper on arXiv