Using one language model to check another's work sounds like a natural safety layer. If a summarizer hallucinates a claim, the reviewer should catch it. A recent experiment found that this assumption holds up poorly in practice. The right model helps, but generic reviewers introduce their own problems.

The experiment setup

Nine council meeting transcripts from the MeetingBank dataset served as source material. A qwen3:8b model summarized each transcript. The summaries contained known errors: claims that twisted or invented facts present in the source. A human labeled every claim across the nine summaries, producing a ground truth set of 35 claims: 23 supported by the transcript, 12 unsupported.

Five general-purpose models then reviewed each summary against its transcript. All ran locally through Ollama with temperature zero, a fixed seed, and a context window of 16,384 tokens on a MacBook Pro M5 Pro with 64 gigabytes of memory. The reviewer received the transcript and the summary and returned a corrected version.

What the reviewers caught

The raw catch rates looked encouraging at first. Larger models caught more unsupported claims: qwen3:8b got 3 of 12, qwen2.5:14b got 6, llama3.1:8b got 9, and both qwen3.8:27b-mlx and qwen3:30b-a3b caught 10 of 12.

The problem emerged when looking at what they removed alongside what they caught. qwen3:30b-a3b caught 10 unsupported claims but also deleted 10 of the 23 supported ones. That is half the true content gone. llama3.1:8b removed 7 supported claims. Even the smaller qwen3:8b wrongly deleted 2. Only qwen3.8:27b-mlx balanced the two sides well, catching 10 unsupported claims while removing just 1 supported claim.

Measuring both directions matters because a reviewer that removes everything looks aggressive and thorough. It catches every error, but it destroys accurate content in the process. A reviewer that removes only obvious errors looks careful, but it lets subtle hallucinations slip through. The tension between these failure modes defines what makes a reviewer useful.

Different reviewers, different blind spots

The models did not catch the same unsupported claims. Only 2 of the 12 bad claims were flagged by every reviewer. One claim slipped past all five models. Most unsupported claims were caught by some models and missed by others. The choice of reviewer directly shaped which errors survived into the final output.

This inconsistency means that swapping one reviewer for another does not just change how many errors get caught. It changes which errors get caught. In a production pipeline, that variability becomes unpredictable behavior.

A model built for the task outperforms generalists

bespoke-minicheck:7b was designed specifically for grounded fact-checking. It takes a source document and a single claim, then predicts whether the source supports that claim. Unlike the general reviewers, it processes one claim at a time and returns a support label rather than rewriting the summary.

The results matched the best general reviewer on catch rate: 10 of 12 unsupported claims removed. It removed zero supported claims. And it finished in 10.2 seconds, compared to 63 seconds for qwen3.8:27b-mlx and 130 seconds for qwen3:30b-a3b. The specialized model was both more precise and faster.

Three strategies to reduce damage

The author tested two alternatives to letting a single model rewrite freely. Flagging instead of deleting keeps true content recoverable. If the reviewer flags a supported claim by mistake, a human can restore it. The tradeoff: a cautious flagger like qwen3:8b lets 75% of unsupported claims go unchecked.

A voting rule across multiple reviewers showed a different tradeoff. When all three reviewers must agree before removing a claim, precision hits 100% but recall drops to 17%: only 2 of 12 unsupported claims get caught. When any single reviewer can trigger removal, 10 of 12 get caught but 7 supported claims also disappear. The two-of-three middle ground removes 4 supported claims while catching half the errors.

None of these voting rules solved the core problem. They shifted the balance between false positives and false negatives without eliminating either.

What this means for production pipelines

A reviewer model is not automatically a safety layer. It can improve an output, but it can also introduce new errors while making the result look cleaner. If you only count what it catches, you miss what it changed, removed, or added by mistake.

The practical takeaway: test any reviewer on a small labeled set before trusting it. Measure both sides. Count what it catches and count what it damages. If the biggest risk is false information reaching users, optimize for catch rate. If the biggest risk is losing important content, optimize for precision. And if you can, use a model trained for support judgment rather than asking a general-purpose model to rewrite and fact-check simultaneously.