Most benchmarks test whether a language model can answer a question. Very few test whether the model knows it does not have enough information to answer and can ask the right clarifying question to get there. A new paper from arXiv, submitted in August 2026, formalizes this gap and quantifies how poorly current models handle it. The work introduces MT-InfoSeek, a controlled evaluation suite of 5,251 problems across mathematics, logic, biology, medicine, and general knowledge, and finds that models consistently underestimate how much information they are missing.
The underspecification problem
The paper frames multi-turn information seeking as a k-underspecified constraint satisfaction problem. The variable k measures how many pieces of information are jointly required to determine a unique answer. A simple math problem might require one missing variable (k=1), while a logic puzzle might require two or three facts to resolve (k=2 or k=3). The higher k goes, the more the model needs to ask before it can answer correctly.
This framing matters because real-world queries are rarely fully specified. A doctor asking an AI for a diagnosis will not always provide every relevant symptom. A developer asking for code help will not always specify the runtime environment, the expected input format, or the edge cases that matter. The ability to recognize missing context and request it before answering is a prerequisite for reliable multi-turn interaction, yet it is barely measured by current evaluations.
What the evaluation suite covers
MT-InfoSeek contains 9,006 task instances drawn from five domains. The problems are designed so that each requires a specific set of missing variables to resolve, and the suite controls for problem difficulty by varying k across instances. This lets researchers measure not just whether models can answer, but whether they can identify what they do not know and formulate the right questions to fill those gaps.
The evaluation measures three axes: what the model asks, when it asks, and how the information it acquires affects the final answer. This three-way split is important because a model might ask the right question at the wrong time, or ask questions that do not actually narrow the answer space, or gather sufficient information but still produce an incorrect answer due to reasoning failures.
Models know they need more but do not know how much more
The core finding is that models recognize when additional information is needed, but they systematically underestimate the quantity. On logical problems where k=2, models under-predicted the degree of missing information about four times as often as they over-predicted it. They would stop asking after one question when two were needed, or ask about irrelevant variables while missing the critical ones.
Models also failed to identify a minimal sufficient set of queries. Even when they did ask questions, they often asked redundant ones or missed the combination of facts that would uniquely determine the answer. When given the true value of k as a hint, models improved only marginally, suggesting the problem is not just about knowing how many questions to ask, but about reasoning over what information is actually needed to constrain the answer.
Query order matters more than models realize
In tasks with ordered dependencies, the sequence in which information is acquired affects the final answer. The paper found that an incorrect query order reduced accuracy even when the model eventually gathered all necessary information. This is a subtle but important failure mode: a model might ask for the right facts but in the wrong sequence, and the resulting reasoning path leads to an incorrect conclusion even though all the pieces were technically available.
This finding has practical implications for multi-turn systems. A chatbot that asks follow-up questions in the wrong order might collect all the right information but still give a wrong answer because the reasoning chain built on early answers was flawed. The evaluation separates this effect by measuring information sufficiency independently from answer generation.
Information seeking is a distinct capability
The paper introduces final sufficiency, a metric that records whether the acquired information determines the target answer independent of whether the model actually generates that answer correctly. This separation reveals differences between models that final accuracy alone does not capture. Two models might produce the same accuracy on a benchmark, but one might be getting lucky with its reasoning while the other is genuinely seeking information effectively.
The authors argue that the ability to seek information over multiple turns is distinct from the ability to generate answers, and that current LLM evaluations do not measure it. A model can score well on question-answering benchmarks while being poor at recognizing when it should not answer at all and should instead ask for clarification. This distinction matters for any deployment where the model interacts with users over multiple turns, which is increasingly the default mode for AI assistants.
What this means for building with LLMs
For developers building multi-turn applications, the paper's results are a reminder that "can answer questions" and "can have a productive conversation" are not the same capability. A model that blindly answers an underspecified question will produce confident-sounding output that may be wrong, while a model that can recognize its own uncertainty and ask for the right clarifying information will be more reliable in practice.
The evaluation suite is a useful tool for anyone building or fine-tuning models for conversational use cases. It provides a structured way to measure whether a model is actually improving at information seeking, rather than just improving at pattern-matching answers from training data. As multi-turn interaction becomes the standard interface for AI systems, measuring this capability will matter as much as measuring raw question-answering accuracy.