A new open-source tool called quant-delta-predictor takes a counterintuitive approach to LLM quantization: rather than predicting which configuration will perform best, it is designed to tell you when it cannot give you a reliable answer at all. Built by developer gracejackson-sudo, the tool evaluates quantization schemes against OpenLLM-style benchmarks and returns a calibrated prediction interval — or a refusal, when the underlying data does not support a confident estimate.
The problem of trusting quantization estimates
Quantization — the process of reducing a model's numerical precision to make it smaller and faster — is a central optimization technique for deploying large language models. But the accuracy cost varies dramatically depending on the base model, the quantization scheme, and the model size. A configuration that works well on one model may degrade performance badly on another, and developers often have to rely on estimates that look precise but are built on insufficient data.
The quant-delta-predictor addresses this by borrowing a technique from statistics known as conformal prediction. The tool takes a base model and quantization configuration as input, then produces an interval estimate for the expected accuracy change on benchmarks, along with a confidence level. The headline result is that the calibration holds: 90.1 percent empirical coverage on unseen checkpoints at a nominal 90 percent confidence level.
The point of refusing to answer
What makes the project notable is not the prediction itself but the refusal. When asked to estimate the impact of 4-bit weights on a sub-2 billion parameter model, the tool measured actual coverage of only 68.8 percent against the 90 percent interval it claims. Rather than returning a number that would look as authoritative as a well-calibrated one, it prints "INSUFFICIENT CALIBRATION FOR THIS COMBINATION," shows the coverage it actually measured and the rows of data it based that on, and stops.
The design philosophy is explicit: a quantization risk estimate is only useful if it will tell you when not to trust it. The cells the tool refuses are exactly the ones where a confident-sounding answer would cause the most harm — misleading a developer into deploying a configuration whose behavior is essentially unknown.
What the data covers and what it does not
The tool is built on 850 rows of evaluation data scraped from 102 RedHatAI model cards hosted on Hugging Face. The dataset spans 38 base checkpoints across 8 model families, 6 quantization schemes, and 16 benchmarks. The underlying methods represented include GPTQ, which uses weight quantization, and SmoothQuant, which applies activation smoothing.
Several important limitations are documented honestly. The model family and size features made out-of-family accuracy worse, not better, so the tool deliberately does not use them. The prediction intervals never exclude zero, meaning the tool cannot definitively tell you that a configuration will hurt performance — only that it likely will not. And the training data comes exclusively from checkpoints Red Hat chose to publish, which means the tool may underpredict damage from a badly-tuned quantization recipe that was never shared publicly.
Sub-2B models, mixture-of-experts architectures, and reasoning-distilled models all fall outside the validated envelope. The tool will refuse rather than extrapolate into those regions.
Design and verification
The project runs with three dependencies — numpy, pandas, and scipy — and requires no network access, API keys, or GPU. The command-line tool ranks quantization schemes and returns each one with a 90 percent interval, the worst loss ever observed, the share of evaluations that lost more than 3 percentage points, and a tier classification. Two cells are refused outright, and four more are blocked from the top tier for resting on fewer than three distinct checkpoints.
The verification infrastructure is unusually strict. RANKING.md, the project's main documentation file, is machine-generated from computed values rather than hand-written. Every quantitative claim carries an invisible tag, and a verification script recomputes each tagged value and fails on any mismatch. The test suite enforces that checks run in CI. An independent verification script, using only the Python standard library, regenerates the headline coverage from raw data and diffs it against the pipeline row by row.
The broader takeaway
The negative result is as important as the headline. Per-model prediction carries almost no signal beyond the quantization scheme average. In other words, knowing which specific model you have does not help much in predicting how quantization will affect it — the scheme itself dominates the outcome. The tool's fitted output is, as its author describes it, a small number table: a calibrated historical baseline rather than a true predictor.
That finding has practical implications. For a developer choosing a quantization scheme, the safest approach may not be picking the scheme with the best predicted outcome on their specific model — it may be picking from among the schemes whose calibrated intervals have been validated against sufficient data, and treating the tool's refusal as a warning rather than an inconvenience.
The project is MIT licensed, runs locally, and is available on GitHub. It does not solve the problem of predicting quantization impact with certainty. But it does solve the problem of knowing when you are being given an answer you should not trust — which, in a field where overconfident estimates can quietly degrade production models, may be the more useful thing.