When LLM Decompilers Recompile More and Preserve Less

When LLM Decompilers Pass Every Test Yet Rewrite Your Code

Decompilation turns compiled binaries back into readable source code, and the stakes are high: security analysts depend on it to find vulnerabilities in malware, reverse-engineer closed-source libraries, and audit firmware. For decades, tools like Ghidra and Hex-Rays produced pseudocode full of placeholders like undefined4 and uVar1, often refusing to compile at all. That was annoying but honest. New LLM-based decompilers, fine-tuned to produce clean, idiomatic C, have shifted the goalposts. Their outputs look like real source, compile cleanly, and pass the shipped input/output tests that come with existing benchmarks. This paper asks: does passing those tests mean the decompiled code actually behaves like the original?

The short answer is no. A function may recompile and pass every shipped test while diverging on other legitimate inputs. Worse, a disclosed vulnerability may vanish from the recompiled code with no visible trace. Neither failure is caught by the benchmarks that currently evaluate these systems.

The Measurement Gap: Why Passing Tests Is Not Enough

Most LLM decompilation benchmarks, including HumanEval-Decompile, ExeBench, AnghaBench, and MBPP, supply short functions with flat scalar signatures and a handful of input/output assertions. The training objectives reinforce this: LLM4Decompile is fine-tuned on a next-token loss over reference source, which rewards textual resemblance without checking behavior. SK2Decompile uses reinforcement learning where the structure reward is zero if the output does not compile and adds a Jaccard overlap of placeholder identifiers otherwise. Neither objective measures what happens on a novel input.

The authors demonstrate this concretely with a decode6 function from ExeBench, which maps a linear index into bands of width 6 followed by a variable-width final band. The shipped test suite contains ten input/output pairs, all with small arguments in the range [0, 124]. On those ten, both SK2Decompile and LLM4Decompile pass. But the small operands never exercise the final band, and none triggers 64-bit overflow. When the authors generate a 100-input stress corpus via fuzzing, SK2Decompile matches only 70/100 and LLM4Decompile only 66/100. Raw Ghidra and Hex-Rays match all 100.

The root cause is straightforward: SK2Decompile collapses the fixed stride 6 and the mutable band width D into a single variable, corrupting both the residual and the horizontal band offset. LLM4Decompile replaces the quotient check with a multiplication-based guard that introduces 64-bit overflow on large operands. Both models produce clean, compilable code that reads naturally and passes the original tests. Both are wrong.

Decompile-Diverge: A Fuzzing-Based Behavioral Oracle

To close this gap, the authors propose Decompile-Diverge, a framework that does not rely on fixed or hand-crafted tests. For each function it: (1) synthesizes a driver that exercises the function through its interface, (2) uses AFL++ to fuzz the reference implementation and build an input corpus, (3) re-executes every decompiled candidate on exactly the same inputs, and (4) compares a digest of the bounded observable post-state: the return value, bytes written, writable globals, and the .data/.bss ranges.

The digest is carefully scoped. Pointer-valued state is relocation-masked because relinking can change heap and stack layout. On the GitHub and CVE tracks, a divergence counts only if it reproduces on four re-runs of both reference and candidate, discarding flakiness from address-space layout or uninitialized memory. This makes the Matched rate an upper bound on behavioral agreement across all valid inputs, while Divergence and Crash Absence rates are lower bounds.

The framework assigns one of three labels to each candidate: Matched (behavior preserved on all inputs), Divergence (a changed output, introduced crash, or hang on at least one input), or Crash Absence (a reference crash that disappears, reported only on the CVE track). Crash Absence takes precedence when both apply.

Eight Systems, Nine Configurations, Three Datasets

The evaluation covers eight systems across nine configurations. The refinement systems include LLM4Decompile, Idioms, SK2Decompile, DeGPT-Qwen, and GLM-5.2-Refine. The end-to-end systems include AutoDecompiler, Nova, SLaDe, and GLM-5.2-E2E. LLM4Decompile and DeGPT refine Ghidra output; the others refine Hex-Rays. GLM-5.2 appears in both roles, reading assembly with no decompilation-specific training.

Three datasets form the evaluation: the four established LLM decompilation corpora (HumanEval-Decompile, ExeBench, AnghaBench, MBPP), a GitHub track of 300 real library functions from 132 libraries across 106 repositories (291 after exclusions), and a CVE track of 287 vulnerable functions from 94 open-source projects. The GitHub track targets functions without released function-level tests. The CVE track preserves original type definitions and helper routines, compiles with AddressSanitizer and UBSan, and includes both vulnerability-triggering and safe inputs. The final input corpus contains 21,119 inputs across CVE functions, averaging 74 per function.

Results: Build Rate Goes Up, Behavioral Agreement Goes Down

On the established corpora, the pattern is consistent: refinement clears more shipped tests while diverging more often on the generated corpus. LLM4Decompile passes 85% on HumanEval-Decompile versus Ghidra's 81%, but diverges on 13-21% of functions against Ghidra's 1-4%. SK2Decompile reaches 25% Divergence on AnghaBench versus Hex-Rays's 0%. The critical column is Div||P: of candidates that pass every shipped test, the percentage that still diverge. It reaches 13% for Nova on HumanEval, 8% for LLM4Decompile on ExeBench, and pooled, every refinement system exceeds its own front end. Across all passers on all four corpora, 4.9% diverge despite passing every shipped test, and 77% of those divergences involve a changed output with no crash at all.

On real code the divergence is starker. LLM4Decompile lifts Ghidra's build rate from 75% to 90% on the GitHub track, while its Matched rate falls from 74% to 62%. A paired per-function analysis shows 137 functions matched under both, 77 under Ghidra alone, 42 under LLM4Decompile alone, and 35 under neither (McNemar p=.002). Recompilability ranks the refiner higher; behavioral agreement ranks the front end higher.

On vulnerable code, the same rewriting becomes a security problem. Ghidra achieves only 30% build rate on CVE functions because project-specific types resist recompilation. LLM4Decompile raises this to 64%, but 25 of its 183 builds lose the reference crash, yielding a track-wide Crash Absence rate of 8.7% (95% CI [6.0%, 12.5%]). Bootstrap leave-one-project-out confirms no single project drives it. Raw Ghidra has zero Crash Absence cases; Hex-Rays's three losses arise from mis-striding or undefined behavior artifacts.

Why the Rewrites Go Wrong: A Source-Level Analysis

The authors trace divergence to a shift from visible front-end placeholders to confident inventions. When a traditional decompiler cannot resolve something, it leaves behind tokens like DAT_*, undefined4, or goto constructs. These are ugly but honest: they signal exactly where analysis failed. When an LLM rewrites this pseudocode into clean C, it must supply fields, types, and callees that do not exist in the binary.

Across 40,345 function-candidate pairs, the authors quantify these changes. Systems whose compilation failures retain front-end artifacts (Ghidra, Hex-Rays, DeGPT-Qwen) keep Divergence at or below 7% of their GitHub builds. Systems that hallucinate new code perform much worse: every configuration where over 40% of compilation failures involve an introduced symbol exhibits Divergence of 20.3% or higher.

The association holds within systems too. Functions whose output carries an introduced field, type, or callee diverge far more often than invention-free functions: 53% versus 20% on GitHub (odds ratio 4.6, Fisher p less than 10^-6) and 59% versus 31% on CVE (p less than 10^-4). The gap persists within function-length terciles, suggesting it is not driven by difficulty alone.

The inventory of inventions is revealing. LLM4Decompile removes most front-end vocabulary but makes small, compilable inventions such as a u32 type, a Mat4 struct, or a changed hexadecimal constant. These tend to compile fine and surface as behavioral divergence. SK2Decompile rewrites aggressively, including target renames, storage-class changes, and introduced fields. Its real-world build rate is roughly half that of Hex-Rays. DeGPT-Qwen edits conservatively, removing only 16% of front-end vocabulary and staying at or below 1% on each invention axis, which keeps its failure profile close to Ghidra's.

Limitations and Trade-Offs

Decompile-Diverge has concrete boundaries. The input corpus is finite, so the Matched rate is an upper bound on behavioral agreement and Divergence is a lower bound on behavioral disagreement. Functions with multi-level pointers, function pointers, multidimensional arrays, by-value structs, or project-specific aggregates require manually written drivers, limiting scale. Some functions cannot be built with AddressSanitizer and are excluded. The fuzzing time budget (5 minutes for established corpora, 10 minutes for GitHub and CVE) leaves some code paths unexplored, though extending the budget did not improve coverage in most cases.

The framework also conflates several failure modes under the Divergence label: changed outputs, introduced crashes, hangs, and modifications to globals all count the same way. A crash-only oracle would miss the 10 of LLM4Decompile's 183 CVE builds that keep the crash but change observable state. The CVE track's Crash Absence definition requires a validated proof of concept that no longer crashes, which is a strong condition that does not capture subtler corruption.

On the systems side, the evaluation is constrained by reproducibility. Several recent decompilation systems with runtime feedback and reward shaping were excluded because public artifacts were unavailable. SLaDe's source-to-assembly pipeline cannot handle header-dependent library functions as standalone units, resulting in 1% build rate. Nova's non-causal attention mask forces a dense tensor that requires 100 GB of VRAM for a 40k-token function, limiting practical deployment.

What This Means for Developers and Security Analysts

If you are using LLM-based decompilers for security auditing, the core lesson is: recompilability does not equal correctness. A clean, compilable output that passes its test suite can silently rewrite vulnerability logic. The Decompile-Diverge framework offers a practical evaluation methodology: synthesize a driver, fuzz the reference, and compare post-states. For anyone training or fine-tuning decompilation models, the paper suggests that next-token loss and structural rewards are insufficient objectives. Behavioral agreement on diverse inputs should be part of the training signal.

For practitioners evaluating decompilation tools, the paper provides a concrete diagnostic: compare the vocabulary of the output against the front-end input and the reference. Introduced fields, types, and callees that do not appear in either are red flags. The more aggressively a system removes front-end placeholders, the more likely it is to introduce behavioral divergence. Conservative editors like DeGPT-Qwen, which keep most front-end tokens intact, achieve Divergence rates close to raw decompilers, suggesting that restraint is a feature, not a limitation.

The paper also raises a design question for future work: should decompilation benchmarks ship fuzzed input corpora alongside their test suites? The shipped tests in HumanEval, ExeBench, and MBPP are too narrow to catch the divergence patterns documented here. A richer input distribution would force training objectives toward behavioral agreement rather than textual similarity.

The Path Forward

Decompile-Diverge establishes that current LLM decompilers are optimizing for the wrong metric. The systems that produce the cleanest, most compilable code are also the ones most likely to silently change function behavior. Source-level analysis connects this to a fundamental shift: traditional decompilers leave visible scars where they fail, while LLMs fill those scars with plausible-looking inventions that may be wrong. The framework does not solve the problem, but it makes the problem visible and measurable, which is a prerequisite for fixing it.