Signing weights does not guarantee running the same model
An October 2026 arXiv preprint reveals a verifiability gap in how quantized LLMs are distributed and used. The paper, titled "What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains," demonstrates that model signing, which authenticates full-precision weights, does not ensure users run the exact artifact they signed. The authors measure what they call the Independent Verifiability Rate (IVR) — the fraction of a popularity-sampled GGUF ecosystem that a third party can re-derive bit-for-bit.
IVR near zero across popular GGUF repositories
The study examined the 200 most-downloaded Hugging Face GGUF repositories, narrowing to 198 in scope. Under a recorded-inputs protocol, the IVR was 0/198: no artifact or model card recorded the build environment. This means a third party cannot re-derive the exact quantized file from disclosed inputs alone. The finding holds even though the artifacts are not strictly irreproducible — the authors found they could re-derive some files when additional coordinates were supplied.
Missing coordinates prevent bit-for-bit re-derivation
Four repositories in the sample disclosed their inputs. The model card's information alone could not reproduce the Q4_K_M file. However, two coordinates the card omitted enabled partial re-derivation: the build's floating-point-contraction setting and, for hybrid models, the per-tensor quantization recipe. With these, the authors re-derived 7/8 target-quant files bit-for-bit on x86 (5/8 on arm64). The fp16 file re-derives from the provider's checkpoint, but the calibration imatrix must be shipped and hash-bound, since recomputing it did not match. An older release additionally pinned the instruction set, and re-derivation failed there.
One compiler setting yields unique output across platforms
The authors show that a single compiler setting produces a unique output across the tested platforms at negligible quality cost — +0.06% perplexity. This finding, reproduced from issue #25353, establishes that the execution environment E is a necessary coordinate for verifiability. The same compiler parameters can be verified across x86 and arm64 builds, though the output diverges by platform.
Security characterization: floating-point noise versus tampering
The paper characterizes benign floating-point non-determinism as separable from uniformly placed nibble-flip tampering above a detection floor that grows with the environments a verifier must allow. However, a targeted embedding edit evades this statistic. The proposed protocol rests on exact re-derivation; adversarial-fp16 attacks lie outside its scope. In other words, while the protocol can detect some forms of corruption, it does not cover all possible attack vectors on quantized models.
Release of audit data and harness
The authors release the audit data and a re-runnable harness so the research community can examine the verifiability gap in their own supply chains. The work does not propose a full fix but maps the current terrain: where signing breaks down, which coordinates matter, and where the protocol has blind spots.
Why this matters for LLM users and distributors
Model signing gives a false sense of continuity. A user who verifies a signature on a full-precision model may unknowingly run a different quantized artifact. The gap arises because the transformation from W to Q is not uniquely disclosed. For teams building on quantized LLMs, the finding suggests that reproducibility requires more than trusting a model card — it requires recording the full build environment, including floating-point-contraction settings, quantization recipes, and compiler flags. The released harness provides a starting point for making supply chains more transparent.