What RTL repair benchmarks claim to measure
RTL repair benchmarks should measure whether a model can fix broken RTL given a specification. When large language models target hardware description languages, the assumption is that a lower repair rate reflects a model's inability to diagnose and correct the flaw. But as Zhu and Ranawat argue, the score a benchmark reports can depend on things the benchmark does not control—reset behavior, first-cycle comparison, and whether the model is actually using the faulty RTL or simply regenerating from the specification.
The authors build two resources. RTLRepairDataset contains 5,784 (broken, fixed) training pairs, constructed through a hash-bound pipeline that ties each pair to a specific source seed. RTLRepairBench holds out 459 cases, 374 with syntactic mutations and 85 with behavioral mutations, all from disjoint seed sets. A companion evaluation harness supports two oracle tiers: Tier S, which runs the candidate and golden module for 200 simulation cycles under sampled stimuli and checks output agreement at every cycle; and Tier F, which applies design-matched equivalence checking under reset, initialization, and unknown-value semantics.
Constructing RTLRepairDataset and RTLRepairBench
The dataset construction proceeds in three gates. First, a synthetic mutation is injected into a Verilog module—syntactic mutations such as dropping an endmodule, deleting a semicolon, unbalancing a begin/end block, or renaming a declaration without updating its uses; behavioral mutations include op_swap, flip_reset_polarity, differential iverilog simulation, flip_equality, swap_ternary, swap_logical, and off_by_one_literal. Second, the builder runs verilator --lint-only to confirm the mutation introduces a lint error, and third, it confirms the "fixed" version in the training pair resolves the error. This tool-gated process ensures that every training pair reflects a genuine, diagnosed repair rather than an arbitrary code change.
RTLRepairBench follows the same mutation taxonomy but holds out 459 cases from source seeds not seen during training. The mutation distribution appendix reports that 374 bench mutants are syntactic and 85 are behavioral. Four semantic operators—op_swap, flip_equality, swap_ternary, and swap_logical—appear only in the bench set, meaning the training data never exposed the model to these failure modes. Decontamination uses seed identifiers from VerilogEval and RTLLM; after normalization, zero overlap exists between 1,895 distinct training seeds and the 130 benchmark seeds.
Oracle definitions: simulation versus formal equivalence
The evaluation harness ships with two oracle types that produce different verdicts on the same candidate. Tier S, the simulation oracle, loads the candidate and the golden module into Verilator, applies 200 randomly sampled stimulus cycles, and records whether the outputs match at every compared cycle. Agreement across 200 cycles does not prove equivalence—it only confirms no counterexample appeared in that trace. Tier F, the formal oracle, applies a contract matched to the design: exhaustive combinational checking for 17 designs, a declared reset sequence for 60, or a designated preamble for 8, followed by equivalence checking under stated initialization and unknown-value semantics. The choice of contract, initialization assumption, and reset heuristic changes whether Tier F succeeds or flags the candidate as inequivalent.
The paper shows that initialization and reset semantics matter because not all state is fully initialized at reset. If a model’s repair leaves flip-flops in an unknown state, a simulation oracle that samples from a known initial state may pass, while a formal oracle that propagates X-values will fail—or vice versa. The harness documents these assumptions so that a reported score can be reproduced with the same contract.
How reset and first-cycle semantics flip simulation-formal scores
The first measurement confound the authors identify appears when simulation and formal scoring give opposite signs for the apparent effect of providing a specification. Restricted to 79 cases with definitive Tier F verdicts in both arms, the descriptive contrast reverses depending on whether cycle 1 is included. When cycles 0 through 199 are scored, specification presence yields a −11.4 percentage point advantage (75.9% passing with diagnostic input vs. 64.6% with specification). When only cycles 1 through 199 are scored, the advantage swings to +7.6 points (91.1% vs. 98.7%). Skipping the first comparison alone explains the sign reversal.
Reset-aware rescoring, which applies the released two-edge reset heuristic, clears 39 of 39 Tier-S-fail/Tier-F-proved outcomes in the 79-case cohort and 12 of 12 and 27 of 30 across the two arms at the full 85-case scale. The authors attribute almost all simulation-formal disagreement to reset and first-cycle comparison semantics. For benchmarks that rely on simulation matches, the reset sequence and the first compared cycle are part of the measurement definition and should be reported and audited.
Specification-only regeneration versus genuine repair
The second confound addresses whether a conventional repair score actually isolates whether the model used the faulty RTL. On 274 mined model failures—cases where the spec→RTL output compiles but fails the task testbench—each model received one of three prompts: broken RTL without the specification, broken RTL with the specification, or the specification alone. The specification-only arm is a necessary diagnostic: if a model’s output under the specification-only arm matches its output under the specification-plus-broken-RTL arm, it is unclear whether the model repaired the RTL or simply generated a new implementation from the specification.
Table 2 in the paper reports point estimates across six models. For every model, specification-only recovery exceeds no-specification recovery. But relative to specification-conditioned repair, the specification-only point estimate is lower for three models and higher for three, with the gap ranging from −8.5 to +18.3 percentage points. GPT-5.5 shows −4.6 points, Claude Opus 4.8 shows −2.6 points, GPT-5.4-mini shows +5.5 points, GPT-4.1 shows −8.5 points, GPT-4o-mini shows +2.2 points, and Claude Opus 4.6 shows +18.3 points. These results do not establish that a model ignored the faulty RTL internally; they show that output correctness alone cannot identify whether success arose from repair or regeneration.
Controls for reproducible RTL-repair evaluation
The authors translate their findings into four concrete controls for anyone building or reporting RTL repair benchmarks. First, separate syntactic from behavioral bugs. Syntactic errors—lint failures that prevent compilation—are easier for models to fix than behavioral bugs that compile cleanly but produce incorrect outputs. Reporting the split lets readers judge whether the score reflects ease of repair or model capability.
Second, state initialization, reset, first-comparison-cycle, and unknown-value semantics must be disclosed. The oracle tier, contract type, reset heuristic, and initialization assumption are not interchangeable; changing any of them can flip the measured outcome, as the simulation-formal disagreement demonstrates.
Third, include a specification-only baseline as a diagnostic for possible regeneration. The specification-only arm does not, by itself, resolve whether a model used the faulty RTL, but its presence alongside the repair score makes the ambiguity explicit and the score’s interpretation more transparent.
Fourth, report oracle coverage and undecided cases. Not every design has a matched formal contract; some designs fall outside exhaustive combinational checking, reset-sequence verification, or preamble-based equivalence. Undecided cases should be counted and reported rather than silently excluded, so that the denominator of any repair rate is transparent.
Together, these controls make the interpretation of a repair score reproducible alongside the score itself. A repair system or benchmark score can nevertheless create false confidence if deployed as a substitute for verification; generated RTL should continue to undergo independent simulation, formal analysis where supported, and human design review.
Limitations and generalizability
The findings are descriptive and limited by the evaluated cohort sizes, a predominantly single-draw setting, and historical model-output snapshots. Six models were examined across 274 mined failures; the panel was not designed for multiplicity-adjusted confirmatory inference. Future work should test repeated samples, a broader model roster, and larger and more diverse RTL designs to assess how well the findings generalize. The released construction code, benchmark, and reproduction materials aim to lower the barrier for follow-up audits.
RTLRepairDataset and RTLRepairBench make it possible to quantify the gap between syntactic and behavioral repair through a released, hash-bound training resource and its builder, a held-out evaluation set, mutation labels, and explicit oracle outcomes. The audit also shows that simulation and formal-equivalence scores can disagree substantially, with reset and first-cycle semantics explaining almost all observed false divergences, and that a correct output need not demonstrate use of the supplied faulty RTL. More explicit oracle contracts and regeneration controls can therefore reduce misleading RTL-repair claims and improve reproducibility.