The Problem: Compile Rate May Not Mean What Researchers Think It Means
Large language models are increasingly being applied to the automated repair of C/C++ security vulnerabilities. When a model generates a candidate patch, the most commonly reported proxy for progress is whether that patch compiles: compile rate. Papers routinely report that their system achieves a certain percentage of compile rate, and higher numbers are treated as evidence of better repair capability.
Om Nepal, Sushant Aryal, Oluseyi Olukola, and Nick Rahimi from the University of Southern Mississippi argue that this practice is scientifically unsound. Through five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs spanning 350M to 6.7B parameters, and three prompting strategies, they demonstrate that compile rate is not merely noisy but systematically misleading in five distinct ways. Their central claim is that compile rate barely reflects model quality and instead measures properties of the evaluation setup, the toolchain, and the dataset.
Background: Why Compile Rate Became the Default Metric
The use of compile rate as a proxy follows from the practical constraints of vulnerability repair evaluation. Real-world vulnerability fixes require executable tests or proof-of-concept exploits to verify that a patch removes the vulnerability without breaking intended behavior. For many function-level, CVE-derived datasets, no such oracles are supplied with each example. The community therefore falls back on cheap, automatic proxies.
Two metrics dominate. Compile rate (equivalent to pass@1 under uniform sampling) measures whether a generated patch builds. CodeBLEU measures textual similarity to the developer-written fix, augmented with abstract-syntax-tree and data-flow matching. Both were designed for whole-program or whole-function generation. Neither was designed to isolate the small edit that a vulnerability fix typically represents.
The prior literature has documented that BLEU-style metrics agree only weakly with human assessments. Recent work has also found that secure-code evaluations can conflate security with functionality, with models achieving security by deleting code rather than by fixing it. What no prior work has done is systematically isolate and quantify the specific ways in which compile rate fails as a metric for single-function vulnerability repair.
The Five Failure Modes
The paper's five controlled experiments each isolate a distinct way an evaluation metric can be unreliable. The experiments are designed to be independent and mutually reinforcing.
Insensitivity to genuine improvement. When the authors doubled the generation budget from 256 to 512 tokens, the generated code improved substantially by multiple reference metrics, yet compile rate barely responded. The metric was effectively blind to a real quality improvement.
Confounding by the measurement setup. Approximately 64% of compile failures were not attributable to the model at all. Instead, they reflected the evaluation harness and dataset artifacts. This share was nearly invariant across three very differently sized models, from 350M to 6.7B parameters. One specific dataset artifact accounted for 15 to 19% of all compile failures: Big-Vul's preprocessing had stripped return types from functions, so models were faithfully reproducing malformed function signatures that they had never seen during training.
Toolchain dependence. The same identical patches shifted by 1.8 to 2.7 times in compile rate depending on a single compiler standard flag, with zero regressions. A metric that swings by a factor of two on the basis of one compiler flag is measuring the toolchain, not the model.
Disagreement with reference similarity. Compile rate ranked the three models in the opposite order to CodeBLEU. The smallest model, CodeGen-350M, achieved the highest compile rate (5.45%) but the lowest CodeBLEU (0.463), while the largest, DeepSeek-6.7B, had the lowest compile rate (3.32%) but the highest CodeBLEU (0.675). The two metrics told opposite stories about which model was better.
Gameability under optimization. When compile rate was used as an optimization target, a compiler-feedback loop raised compile rate while similarity to the human fix fell. Manual inspection found deletion- and placeholder-style non-repairs among the newly compiling outputs. The model learned to produce code that builds, not code that fixes the vulnerability.
The Natural Fallback Also Fails
When compile rate fails, the natural fallback is whole-function CodeBLEU. But the paper shows this metric is itself gameable. An unchanged copy of the vulnerable input, which changes nothing, outscores every model tested. The reason is structural: most of a correct fix is unchanged surrounding context, so preserving that context while making no repair earns a high CodeBLEU score. The metric rewards the absence of editing rather than the presence of a correct fix.
A Change-Aware Screen: diff_F1
Motivated by these failures, the authors examine diff_F1, a change-aware screen that scores only the edited region rather than the whole function. It is computed on a line-level diff, comparing only the lines added and removed during the repair, and is defined as the average token-level F1 between the developer's edits and the model's edits over the sides where changes occurred.
The results are promising but qualified. A no-op patch, which changes nothing, receives exactly zero credit, as intended. Some deletion-based gaming patches observed in the optimization experiment also receive near-zero credit. However, diff_F1 still credits some deletion-style patches and does not catch all of them. It does assign nonzero credit to model edits that genuinely overlap the developer's edit, which is a meaningful signal.
The authors are careful to frame diff_F1 as a screen, not a repair-quality metric. It may serve as a cheap pre-filter before deeper, execution-based analysis, but it has limitations: its credit for certain deletion-style patches can be considerably higher than intended, and it does not verify that the edit is functionally correct.
The Bigger Picture
The paper makes a broader methodological point about the unit of evaluation. Prior repair and code-generation work scores the generated output taken as a whole, whether by unit tests, reference similarity, exact match, or a static analyzer. None of it scores the edit that a vulnerability fix actually consists of. Because a vulnerability fix is typically a small modification to a function, evaluating the whole function conflates the repair with all the unchanged surrounding code.
The authors also identify a concrete data-quality issue in Big-Vul worth reporting on its own. In 96 to 100% of cases where functions fail to compile due to missing return types, Big-Vul's own preprocessing had already stripped the return type from the stored function. The model is faithfully reproducing a function signature that was malformed in the input. This means a non-trivial share of failures attributed to models in compile-rate studies based on Big-Vul is actually inherited from dataset preprocessing.
Limitations
The study has several limitations. The test set covers seven CWE categories from Big-Vul but does not survey every category in the dataset. The evaluation uses an isolated-function compile harness, which, while carefully designed, cannot fully replicate the complexity of a real project context. The three models tested are all base (non-instruction-tuned) code LLMs, so results may not generalize to instruction-tuned models or larger systems. The diff_F1 metric is explored as a screen but is not validated as a comprehensive repair-quality measure. The paper also acknowledges that static analysis via Semgrep was degenerate in this setting, returning no findings across 9,131 patches, which highlights the difficulty of automated vulnerability verification in isolation.
What This Means in Practice
For researchers working on LLM-based vulnerability repair, the paper's findings have immediate practical implications. Compile rate, the most commonly reported metric in the literature, should not be interpreted as a measure of repair quality or model capability. The fact that 64% of compile failures are attributable to the evaluation setup rather than the model means that reported compile-rate numbers are confounded by factors outside the model's control.
The paper argues for a shift toward change-aware, execution-grounded evaluation. Change-aware metrics like diff_F1 score the edit rather than the whole output, and execution-grounded evaluation uses actual tests or formal verification rather than syntactic proxies. While building such evaluation harnesses at scale is challenging, the paper demonstrates that cheap change-aware screens can already filter out the worst failures.
The paper is 23 pages with 11 tables and a comprehensive experimental design. It does not propose a new repair model or a new evaluation benchmark. It proposes a critique and a partial remedy. Its value lies in forcing the community to reconsider what its metrics actually measure, and in providing the evidence to support that reconsideration. The code and data are publicly released at https://github.com/OmNepal/llm-vulnrepair-metrics, making the experiments reproducible.