The problem of evaluating large language models becomes acute when the questions asked have no single correct answer. Policy worth, option selection, competing values—these are the everyday questions put to LLMs that resist traditional accuracy scoring. Stated-preference economics has addressed this same problem for decades. It evaluates survey responses without knowing the true value, using a framework of validity and related concepts: content validity, construct validity, criterion validity, reliability, incentive compatibility, and consequentiality. This paper argues that this framework generalizes to LLM evaluation and sets out what each concept means in that context. The approach is demonstrated using a water-quality stated-preference economic valuation survey administered to six models. In this economic application, validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of $75,000, and the two newest pass every test of theoretical validity that can be scored, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.
Evaluating LLMs Without Ground Truth
Most LLM evaluation today relies on benchmark datasets with reference answers. When a question has a definitive correct answer, accuracy, F1, or exact match provides a clean signal. But many real-world questions do not have this property. What is a policy worth? Which option should a user choose? How should competing values be weighed? These questions are open-ended, subjective, or value-laden. In these settings, a model cannot be scored as right or wrong against a ground-truth answer. Yet these are precisely the questions increasingly put to LLMs in practice.
Stated-preference economics emerged to evaluate survey responses about preferences, values, and policy support when the "true" value is unobservable. Its framework has persisted for decades precisely because it does not require ground truth. The author argues that this framework is a general method for evaluating LLMs in the absence of correct answers, and they set out to make explicit what each validity concept means for LLM evaluation.
Validity Concepts and What They Mean for LLM Evaluation
The paper maps six validity concepts from stated-preference economics onto the LLM evaluation problem. Content validity asks whether the model's responses cover the relevant domain of the question. Construct validity asks whether the model's responses comport with the theoretical construct the question is intended to measure. Criterion validity asks whether the model's responses predict an external criterion reliably. Reliability asks whether the model gives consistent responses across repeated exposures. Incentive compatibility asks whether truthful responses are the model's dominant strategy. Consequentiality asks whether the model's responses would meaningfully change if the stakes or context were altered.
For each concept, the paper explains what a passing test signals and where the test falls short. Content validity does not guarantee truth; it only gauges coverage. Construct validity does not guarantee correctness; it gauges theoretical alignment. Criterion validity does not guarantee correctness either; it gauges predictive power. Reliability does not guarantee correctness; it gauges consistency. Incentive compatibility does not guarantee correctness; it gauges whether the model's optimal strategy is to report truthfully. Consequentiality does not guarantee correctness; it gauges whether the model's behavior is sensitive to context changes. Across all six, a passing validity test signals that the model's answers are coherent and theoretically plausible, not that they are factually correct.
An Economic Application: Water-Quality Stated Preference
The paper demonstrates the framework using a published water-quality stated-preference economic valuation survey (Vossler et al., 2023) administered to six LLMs. The survey asks respondents about their willingness to pay for improvements in drinking-water quality, varying the scope of the improvement and the household income level. These variations generate testable predictions from economic theory: demand should slope down (as price increases, willingness to pay decreases), and willingness to pay should increase with the scope of the good and with household income.
The six models tested include older and newer generations. At a household income level of $75,000, two older models fail the most basic validity test—their willingness-to-pay responses do not slope down as theory predicts. The two newest models pass every theoretical validity test that can be scored. However, the two newest models diverge on convergent validity, which asks whether the model's willingness-to-pay estimates align with those from other independently developed economic models or empirical datasets.
The results show that the validity tests separate the models sharply. The older models fail basic theoretical predictions; the newest models pass them all but differ in how their estimates converge with other economic evidence. This separation demonstrates that the framework can distinguish models that fail to respect basic economic rationality from those that do, and even among passing models, it can reveal meaningful differences.
What Passing Validity Tests Shows
The paper emphasizes throughout that passing validity tests shows that a model's answers are coherent, not that they are correct. A model can have demand that slopes down and willingness to pay that responds appropriately to scope and income while still giving economically nonsensical or factually wrong answers on specific items. Validity is a necessary but not sufficient condition for correctness. This distinction is central to the paper's contribution: it offers a way to evaluate LLMs when correct answers are unavailable, with the explicit caveat that validity does not imply truth.
Limitations and Trade-Offs
The framework relies on the availability of theoretically grounded predictions. In domains where no such predictions exist, the validity approach cannot be applied. The water-quality survey provides specific testable claims about demand slopes and income effects; other applications may require different theoretical predictions, and the paper does not provide a generic catalogue of such predictions. The tests are necessary but not sufficient for correctness; a model can pass all validity tests and still produce factually erroneous answers on individual items. The experiment administers the survey to only six models; broader model coverage would strengthen the generalizability of the findings. The economic setting is specific; it is unclear how well the validity concepts translate to other question types such as creative writing, code generation, or mathematical reasoning without adaptation. The survey instrument itself is fixed; the framework does not address how to derive validity tests for arbitrary new question domains.
What This Means for LLM Practitioners
For practitioners evaluating LLMs in settings without ground truth—policy analysis, value assessment, multi-option comparison—the stated-preference validity framework offers a practical starting point. The first step is to identify testable predictions from the relevant theory. For economic questions, these are demand slopes, scope effects, and income effects. For other domains, the practitioner must articulate the theoretical expectations the model should satisfy. The second step is to design or select a survey or prompt set that elicits responses capable of being tested against those expectations. The third step is to score the model's responses against the validity tests and interpret the result as a signal of coherence, not correctness. The paper's experiment shows that this scoring can separate older from newer models and even distinguish among newer models on convergent validity, which may inform model selection or improvement priorities.
Developers building LLM-based decision aids should note that validity testing does not replace the need for independent verification or human judgment. A model that passes all validity tests may still produce incorrect or misleading answers on specific items; the tests confirm internal coherence and theoretical alignment, not factual accuracy. The framework is most useful as a diagnostic: if a model fails validity tests, it is likely misaligned with the intended theory; if it passes, it is likely aligned, but additional checks are still needed for correctness.
For researchers, the paper opens a line of work on deriving theoretically grounded validity tests for LLM evaluation beyond the economic domain. The framework's six concepts provide a vocabulary for thinking about what it means for a model's responses to be valid in the absence of ground truth. Future work could catalogue domain-specific predictions, test the framework on larger and more diverse model sets, and explore how validity and correctness interlock in specific applications.
Read the paper on arXiv