Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of which can tell whether a new scientific model is valid. The AI Science Exam for El Nino-Southern Oscillation (SciExam for ENSO) is a benchmark in which agents build low-order stochastic models of ENSO, the dominant mode of interannual climate variability, from real observations. Within a six-hour budget, agents process the observations, write their own diagnostics, which are then frozen, and develop a model using only these diagnostics as feedback. Hidden graders then test whether the model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, and score a published model in the same way. Across twelve agent systems, six produce models that score higher than the published model, mainly through better reconstruction and forecasting. The simplified forms of the stronger models are each compatible with one of the two competing explanations of ENSO's warm-cold asymmetry, an open debate that the task never mentions. Controlled runs of the top system under varied information suggest that its scores do not come from recalling the dated observational record and that the information it receives shapes how it builds its model. SciExam for ENSO can thus evaluate agent research where no answer is known, and the results suggest that agents can already build competitive models whose structures bear on questions that scientists still debate.
Why existing evaluation fails open scientific research
Language-model agents are increasingly asked to carry out open-ended scientific research, yet the standard evaluation paradigm offers no mechanism for assessing whether a newly proposed model constitutes genuine scientific progress. Results are typically graded against a known answer, a rubric, or a language-model reviewer, and each of these approaches has fundamental limitations. A known answer assumes the research question has already been resolved, which defeats the purpose of open-ended inquiry. A rubric requires human experts to anticipate all possible failure modes and successes, a practical impossibility for novel scientific domains. A language-model reviewer risks circularity, rewarding answers that mimic the training distribution rather than advancing understanding. None of these methods can determine whether a new scientific model is valid, because validity in science depends on whether the model reproduces empirical phenomena, predicts unobserved variables, and endures falsification attempts — criteria that none of the common evaluation approaches capture.
The AI Science Exam for ENSO
To address this gap, the authors introduce the AI Science Exam for El Nino-Southern Oscillation (SciExam for ENSO). ENSO, the El Nino-Southern Oscillation, is the dominant mode of interannual climate variability and one of the most studied problems in climate science. Yet it remains contested: the precise mechanisms that generate ENSO's warm-cold asymmetry — why El Nino events tend to be stronger and more prolonged than La Nina events — are still debated. SciExam for ENSO tasks language-model agents with building low-order stochastic models of ENSO from real observations, without providing a target model or answer key. The benchmark is designed so that hidden graders can test whether the model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, offering a validity assessment that does not rely on a pre-specified answer.
Six-hour budget and the agent workflow
Within a six-hour budget, agents process the observations, write their own diagnostics, which are then frozen, and develop a model using only these diagnostics as feedback. This workflow mirrors how human researchers operate: they explore a dataset, derive diagnostic quantities that capture essential features, and then build parsimonious models that explain those diagnostics. The six-hour budget constrains the depth and breadth of exploration, forcing agents to prioritize which diagnostics to compute and what model class to adopt. Crucially, the diagnostics are not provided — each agent must discover them independently — and the model must be built using only those self-generated diagnostics as feedback signals.
Results across twelve agent systems
Across twelve agent systems, six produce models that score higher than the published model, mainly through better reconstruction and forecasting. This is a striking result: even without a target model or answer key, a majority of agents can build models that outperform the published ENSO literature on the benchmark's validity criteria. The stronger models achieve higher scores not by overfitting to the training observations but by improving on the two core metrics that hidden graders assess: reconstruction of known variables and forecasting of held-out years. These results suggest that the task is solvable by current agent architectures and that the bottleneck is not agent capability but the design of the evaluation framework itself.
Models that speak to open scientific debate
An unexpected and scientifically significant finding is that the simplified forms of the stronger models are each compatible with one of the two competing explanations of ENSO's warm-cold asymmetry, an open debate that the task never mentions. Two primary explanations have been proposed in the climate science literature: one attributes the asymmetry to oceanic dynamics, specifically the role of mean-state background conditions in the tropical Pacific; the other attributes it to atmospheric dynamics, specifically the nonlinear interaction between wind bursts and oceanic recharge processes. The fact that independent AI agents, tasked only with building a model from observations, produce models whose simplified structures align with either explanation suggests that the observational data contains sufficient information to distinguish between these mechanisms, and that current agent architectures can discover this distinction without being explicitly told about the debate.
Controlled runs rule out data memorization
Controlled runs of the top system under varied information suggest that its scores do not come from recalling the dated observational record. The top system's performance is sensitive to the specific information it receives: when given different subsets or permutations of the observed data, it builds measurably different models, and its scores change accordingly. If the system were simply memorizing the observational record, such variation would not produce systematic score changes — the memorized record would yield consistent performance regardless of input ordering. Instead, the information it receives shapes how it builds its model, indicating that the agent is performing some form of inductive reasoning from the data rather than retrieving cached observations.
SciExam for ENSO as a general framework
SciExam for ENSO can thus evaluate agent research where no answer is known, and the results suggest that agents can already build competitive models whose structures bear on questions that scientists still debate. The benchmark opens a new paradigm for evaluating AI scientific capability: rather than grading against a known answer, the evaluation tests whether the model survives validation against empirical criteria — reproduction of statistics, recovery of unobserved variables, and forecasting of held-out data. This paradigm is not specific to ENSO; it can be applied to any domain where real observations are available and a low-order model can be built to explain them. Potential domains include planetary climate variability, geological processes, ecological dynamics, and even astrophysical phenomena where observational data exists but a first-principles model is lacking.
Implications for AI-driven scientific discovery
The results have several implications for the future of AI-driven scientific discovery. First, they demonstrate that language-model agents can transcend the role of summarizing existing knowledge and instead generate novel scientific models. Second, the benchmark provides a concrete, reproducible framework for assessing whether a model is scientifically valid, addressing a long-standing gap in AI evaluation. Third, the connection to the ENSO warm-cold asymmetry debate suggests that AI agents can contribute to unresolved scientific questions, potentially offering new perspectives or distilling insights from data that human researchers have missed. Fourth, the fact that six of twelve agents outperform the published model on validity criteria indicates that current agent architectures have significant scientific capability, and that improvements in agent design, data processing, or model class could yield even stronger results.
Conclusion
SciExam for ENSO demonstrates that language-model agents can build low-order stochastic models of a complex climate system from real observations, and that a majority of agents produce models that score higher than the published literature on validity criteria that do not rely on a known answer. The simplified forms of the stronger models align with competing scientific explanations of ENSO's warm-cold asymmetry, suggesting that the observational data contains mechanistic information that agents can discover. Controlled runs rule out simple data memorization, showing that the information received shapes how the model is built. SciExam for ENSO provides a framework for evaluating agent research where no answer is known, and the results indicate that agents can already contribute to open scientific debates through model building.