LLMs Should Know When to Say "I Don't Know"

Language models are fluent by default. Ask them a question they cannot answer, and they will produce something that sounds right, reads well, and might be completely wrong. In healthcare, law, and finance, this is a problem. A confident but unsupported answer can do more harm than silence, and a system that always commits to an answer cannot distinguish the questions it knows from the questions it does not.

Ali Senol from Tarsus University proposes a prompt-only framework called Chain-of-Self-Questioning (CoSQ) that makes answer commitment conditional on an explicit assessment of the information required to answer a question. The system asks the model to identify what it needs to know, evaluate how well it knows it, and then decide whether to answer or abstain. It requires no fine-tuning, no architectural changes, and no access to model internals. It works through structured prompting alone.

The Core Idea: Assess Before You Commit

CoSQ is a three-stage prompting framework. First, the model identifies the information units needed to answer the question. Second, it evaluates the support for those units, assigning a confidence score between zero and one to each. Third, a decision rule determines whether the model is allowed to commit. If the confidence is above a threshold, the model answers using the accepted information. If it falls below, the model abstains.

The framework has three variants, each with a different decision rule. Grounded-CoSQ uses the mean confidence across all information units. The model commits when the mean score meets or exceeds a threshold tau, and abstains otherwise. If the gate is passed, the final prompt receives only the accepted information units, which reduces the chance that the answer is generated from an explicitly rejected premise.

Critical-CoSQ extends the information-unit stage by asking the model to label each unit as either critical or supporting. Only critical units participate in the gate. This reduces the mechanical effect of the number of supporting details and tests whether selective answering improves when the decision rule focuses on information without which a correct answer is impossible.

Adaptive-CoSQ retains role-aware information units and applies three simultaneous checks: the mean confidence across all claims must meet tau, the mean confidence across critical claims must be at least 0.65, and the minimum critical-claim confidence must be at least 0.40. After acceptance, claims with confidence at least 0.40 are passed to the answer stage. The answer is generated in a confident mode when overall confidence is at least 0.75 and in a cautious mode otherwise, then checked for contradiction; a detected contradiction produces an abstention.

The Setup: Eleven Models, Eighteen Conditions

The evaluation uses the 817-question validation split of TruthfulQA in its multiple-choice configuration. To remove fixed-position structure, option order is deterministically balanced with a fixed seed. A response is correct when its selected label matches the balanced gold label. Explicit CoSQ abstention is a separate outcome and does not enter the denominator of answered accuracy.

The model panel contains eleven instruction-tuned or hosted families: Llama 3 (8B and 70B), Llama 4 Scout (17B), Gemma 3 (12B), Gemma 4 (31B), Mistral (7B), GPT-OSS (20B and 120B), GPT-5.5, Claude 5 Sonnet, and DeepSeek Flash. Each model is evaluated under seventeen conditions: Direct and CoT baselines, plus Grounded-CoSQ, Critical-CoSQ, and Adaptive-CoSQ at five threshold values from 0.50 to 0.90.

The metrics track answered accuracy (correct committed answers divided by total committed), coverage (fraction of questions answered), and hallucination rate, defined here as the unconditional wrong-commitment rate: the proportion of all questions for which the model commits to an incorrect option. This is not a complete linguistic annotation of every type of hallucination, but it measures the specific failure mode that matters most in high-stakes settings.

The Numbers: 32% Fewer Wrong Answers

At the primary operating point of tau equals 0.90, Grounded-CoSQ reduced the mean unconditional wrong-commitment rate from 13.1% under chain-of-thought prompting to 8.9%, a 32.1% relative reduction. Answered accuracy increased from 86.9% to 89.7%, while the system answered 87.6% of questions. Both improvements held for all eleven models and at every evaluated threshold.

Critical-CoSQ provided the broadest coverage at 88.6% of questions, with a wrong-commitment rate of 9.2% and answered accuracy of 89.6%. Adaptive-CoSQ was the most conservative, covering 86.5% of questions with the same 8.9% wrong-commitment rate as Grounded-CoSQ and 89.6% answered accuracy.

The abstention composition analysis quantifies what each policy refers for review. Grounded-CoSQ referred 100.5 questions per model on average. Of those, 31.3% were correct abstentions, meaning the baseline would have answered them incorrectly. The remaining 68.7% were lost value, abstentions on questions the baseline would have answered correctly. These are not scoring errors. They quantify the reduction in automated service breadth associated with selective commitment.

The threshold sweep shows that every one of the fifteen selective configurations had lower hallucination rate than the chain-of-thought baseline, although improvement was modest at lower thresholds and largest at tau equals 0.90. The sweep exposes a family of operating points from which a system designer can choose according to the relative consequences of wrong commitments and abstentions.

Why This Works Across Models

The aggregate result is not driven by a single model. At tau equals 0.90, all three CoSQ variants had lower wrong-commitment rates and higher answered accuracy than chain-of-thought for every model in the panel. Grounded-CoSQ hallucination-rate reductions ranged from 1.84 to 9.79 percentage points across models. DeepSeek Flash showed the largest absolute reduction, from 17.7% to 8.0%. Llama 3 8B showed the smallest reduction, from 19.3% to 12.1%, but still improved.

Larger or newer models did not automatically produce better CoSQ performance. Model families differed in how their self-assessed confidence scores mapped to the acceptance gate, but the risk reduction was directionally stable. This argues for keeping selective answering as an explicit system component even when the underlying model is strong.

The paired statistical analysis confirms this. Mean hallucination-rate reduction was 4.22 percentage points with a 95% bootstrap interval of 2.72 to 5.95 points. Mean answered-accuracy increase was 2.87 points with a 95% interval of 1.77 to 4.18 points. Both directional Wilcoxon tests yielded p equals 0.00049, and all eleven model-level differences had the hypothesized sign.

Generalization to Open-Form Questions

A secondary 300-question Natural Questions Short-Answer evaluation provides convergent evidence that selective risk control transfers beyond TruthfulQA-MC. Grounded-CoSQ reduced the mean hallucination rate from 41.2% under chain-of-thought to 27.4% while retaining 83.0% coverage and increasing answered accuracy from 58.8% to 67.0%. The improvement was consistent across the five models tested.

The NQ-Short results should not be compared numerically with the primary MC estimates because it uses open-form generation and semantic answer matching, but the directional pattern is the same: selective answering reduces wrong commitments and improves conditional accuracy when the model has sufficient support.

The Cost of Asking Before Answering

CoSQ adds inference calls because information extraction and confidence assessment precede answer generation. The framework requires three prompt-completion interactions per question for Grounded and Critical variants, and three to four for Adaptive. This overhead is a deliberate exchange: the system spends additional inference budget to reduce the probability of a wrong committed answer.

In high-stakes settings, this trade-off makes sense. A physician who refers a case to a specialist when evidence is insufficient has not failed to provide care. The referral is a responsible decision that limits the risk of an incorrect diagnosis. CoSQ applies the same operational principle to language-model answers. When the evidence for a commitment is insufficient, abstention can be preferable to a fluent but unsupported response.

What the Threshold Actually Means

The confidence threshold is not a calibrated probability. It is an empirical operating parameter that determines a point on a risk-coverage frontier. The paper is explicit about this: threshold values are not literal probabilities, and the optimal value depends on the relative cost of wrong answers versus abstentions in the specific deployment context.

The complete threshold sweep from 0.50 to 0.90 shows that all fifteen selective configurations outperformed chain-of-thought on hallucination rate. But the improvement was modest at lower thresholds. The sweep does not imply a single universally optimal threshold. It exposes a design space.

For applications where an unsupported answer is more costly than escalation, the conservative point at tau equals 0.90 is justified. For applications that value broader automated service, a lower threshold or Critical-CoSQ provides more coverage with still-meaningful risk reduction. The framework makes this choice explicit rather than leaving it to chance.

Limitations and What Comes Next

Self-evaluated confidence is not independent. The confidence score comes from the same model that produces the answer. It is not an independent probability estimate and may be miscalibrated. The threshold values are empirical operating points, not calibrated probabilities, and threshold transfer across datasets and domains is unresolved.

The primary evaluation uses TruthfulQA-MC, which has 817 questions. Larger multi-domain evaluations are needed before claiming broad deployment validity. The forced-choice multiple-choice protocol eliminates answer-parsing confounds but does not capture every property of real-world factual QA.

No prompt optimization was performed per model. The study uses a common prompt protocol for comparability, which may leave performance on the table for some models. Prompt robustness across paraphrases, instruction order, and alternative confidence wording is an open question.

The study does not include human evaluation of whether abstentions are appropriate or whether answers are genuinely useful. This is flagged as an important next step. CoSQ should not be deployed in high-stakes settings without domain-specific validation, evidence access, and human oversight.

A Practical Tool for Selective Reliability

CoSQ demonstrates that self-assessment can support explicit, tunable answer-or-abstain decisions when an unsupported commitment is more costly than referral or review. The framework is prompt-only, model-agnostic, and deployable on any LLM endpoint. It adds inference cost but reduces wrong commitments by roughly a third while improving conditional accuracy.

The three variants serve different use cases. Grounded-CoSQ provides the best overall balance. Critical-CoSQ preserves the most coverage. Adaptive-CoSQ provides the most conservative operating point. System designers choose where to operate on the risk-coverage frontier based on their specific cost structure.

The core insight is that LLM evaluation should ask two questions: how accurate are the answers given, and how often does the model decide to give an answer? CoSQ makes this distinction operational, and the results show that the answer to the second question can be meaningfully improved through structured prompting alone.

Read the paper on arXiv