Measuring a scientist's ability to explore unknown territory is straightforward: you observe whether they discover something genuinely new. Measuring an AI system's ability to do the same is extraordinarily difficult. How do you verify that a system has made a genuine discovery rather than simply recalling related knowledge from its training data? A team of researchers from Fudan University, Tencent, and Tsinghua University — behind the ExplorationBench framework — has built what they call "verifiable alien worlds" to answer this question, creating a benchmark where the rules of reality are deliberately alien and every answer can be checked with exact certainty.
The Measurement Problem
Scientific exploration requires three things: deciding what evidence to gather, collecting that evidence through interaction with the environment, and applying what was learned to new problems. Current large language models perform well on static evaluations of knowledge, mathematics, and professional tasks, but these tests primarily measure whether a model can retrieve and reason with information it already has. They do not measure exploration — the capacity to discover and apply genuinely new knowledge.
The evaluation challenge is a tension between two requirements. The tasks must be new to the system so that success cannot come from pre-training recall, yet the answers must be fully known to the evaluator so that success can be verified. Established domains like mathematics and coding meet the verifiability requirement but not the novelty requirement — a model may reproduce what it memorized. Genuinely novel outputs meet the novelty requirement but not the verifiability requirement — verifying a new scientific hypothesis may require expert proof checking or years of observation.
The ExplorationBench team resolves this tension by constructing executable environments whose governing rules deliberately conflict with what any reasonable agent would already know. The rules are executable, so every answer can be checked exactly by an interpreter or proof-checker. They conflict with familiar knowledge, so recall alone cannot solve the tasks. The system must explore the environment to discover the true rules before it can apply them to unseen problems.
The Two Alien Worlds
ExplorationBench contains two sandboxes, each with a flawed manual that describes the standard semantics, which are false for the perturbed parts of the environment. Every system starts from the same flawed manual and the same fixed worked examples. The worked examples establish a baseline — before any exploration, a system's held-out accuracy on AlienCode never exceeds 15.7%, and on AlienLogic it ranges from 32.9% to 51.9%, reflecting that some standard rules remain intact.
AlienCode is a small programming language whose familiar-looking operators follow hidden semantics. Integer literals are silently XOR-ed with 27, so EMIT(100) prints 127 rather than 100. PLUCK counts positions from one even though the manual says zero. CARVE slices operate on shifted indices. The sandbox contains 31 discovery targets and 70 held-out tasks, including base tasks, nested composition tasks, ceiling tasks, and rule-coverage tasks designed to exercise all 31 discovery targets. Programs are graded by an interpreter on private evaluator inputs, so success requires rule-aware executable behavior rather than memorizing displayed examples.
AlienLogic is a first-order natural-deduction system with 24 discovery targets, each a patched inference rule. A proof-checker verifies every submitted proof, and designated unprovable tasks receive credit only when the system correctly declines to prove them. The proofs must be formally valid under the hidden rules, which the system must infer from environmental feedback.
Each sandbox provides a task-specific tool-call schema. In AlienCode, a probe is a candidate program and the feedback is the exact program output. In AlienLogic, a probe is a candidate proof and the feedback is a compact verifier result. The environments are deterministic — the same input always produces the same output — which is what makes exact verification possible.
How the Benchmark Measures Exploration
The protocol is structured around milestones. Every system starts from the same worked examples and makes no tool calls at M0, so all systems begin with identical evidence. The system then explores for four rounds. In each round, it issues tool calls whose arguments are programs or proofs, and the environment executes them and returns deterministic feedback. At each milestone M1 through M4, the system is tested in a separate copy of the conversation with tools disabled. It reports the rules it believes hold and answers the 70 held-out tasks. The copy is then discarded, so testing never adds evidence to the exploration.
Each system runs three independent trajectories per sandbox. The primary score is Best@3, the best final accuracy among the three trajectories. This design choice reflects the researchers' finding that exploration is highly variable — different trajectories of the same system under the same budget can end up to 72.8 accuracy points apart.
The benchmark records three connected parts of the process separately: probe selection (which experiments the system chooses to run), reported discovery (what rules the system believes it has found), and rule use (whether the system can apply what it learned to unseen tasks). This separation is crucial, because the researchers find that discovering a rule and using it are distinct capabilities that come apart in practice.
Key Findings
The evaluation of ten frontier AI systems reveals several important patterns about how current systems explore unfamiliar environments.
Exploration, not recall, produces the knowledge required. After four rounds of autonomous exploration, the best AlienCode trajectory reaches 87.6% held-out accuracy, and 7 of 10 systems exceed 60%. Without-tool answering — adding the same number of model turns but without environment feedback — leaves AlienCode systems at 0.5% to 11.0%. Additional reasoning without evidence is not a substitute for probing the environment.
Who designs the experiments matters. The benchmark compares autonomous exploration, where the system designs each experiment from its own history, against hindsight exploration, which replays the probes of the system's best trajectory, and fixed-probe exploration, which issues a model-independent probe sequence. In AlienCode, autonomous exploration achieves a median of 66.0% against hindsight's 40.7% and fixed probes' 5.7%. The sandbox is deterministic, so the evidence is identical — the gap reflects the value of designing the experiments rather than merely receiving their results. Hindsight exploration beats autonomous for only one system, Gemini 3.8 Flash. In AlienLogic, designing the experiments adds nothing; what matters is which proofs are tried.
Exploration ability does not transfer between domains. The Spearman correlation between the Best@3 rankings in AlienCode and AlienLogic is only 0.35. Grok 4.6 ranks fifth in AlienCode and first in AlienLogic. Gemini 3.8 Flash ties for third in AlienCode and ranks last in AlienLogic. The sandboxes separate systems in different ways — AlienCode spreads Best@3 over 12.9% to 87.6%, while eight of 10 systems fall within 72.4% to 83.8% in AlienLogic.
Discovering rules and using them are separate capabilities. In AlienCode, being told the complete rule set before exploration (O@M0) achieves 84.3% accuracy, but the best autonomous trajectory reaches 87.6% — exploration beats being told the answers. More strikingly, when a trajectory's final rule report states every rule a task requires correctly, the task is still solved only 70.9% of the time. Knowing a rule does not guarantee using it correctly. Two off-by-one rules — PLUCK's index shift and CARVE's slice shift — together enter 51 of the 70 AlienCode tasks, and 13 of the 15 trajectories that end at or above 50% state both correctly. When these keystone rules are discovered, accuracy jumps coincide with their discovery, but the rule report itself is a lossy readout of what the system can actually do.
In AlienLogic, being told beats exploring. The complete rule set supplied before exploration achieves 93% to 97% accuracy, and no Best@3 trajectory matches it. Exploring first adds nothing (A4+O minus O@M0 has a median of 0.0 points). AlienLogic is limited by discovery — finding the hidden inference rules — while AlienCode is limited by use. This asymmetry suggests that different domains require fundamentally different exploration strategies.
Exploration is unreliable. Trajectories of the same system under the same budget end up to 72.8 accuracy points apart. Six of the 30 AlienCode trajectories end at least 3 points below an earlier milestone, meaning continued exploration can stall or reverse earlier gains. The variability far exceeds the noise from repeated answering to the same questions, suggesting that exploration success depends heavily on chance in which probes the system happens to try first.
What This Means for AI Research
ExplorationBench provides a concrete, tractable framework for measuring a capability that has been difficult to quantify: the ability to acquire and apply genuinely new knowledge through exploration. The benchmark's design — executable environments with rules that contradict familiar priors — offers a clean separation between recall and discovery that static benchmarks cannot provide.
The findings carry implications for how we think about AI systems entering scientific and engineering workflows. Frontier systems can acquire unfamiliar rules through exploration, but they do so unreliably. They design their own experiments effectively in some domains but not others, and knowing a rule does not guarantee being able to use it. The high variability between trajectories of the same system suggests that current exploration methods depend substantially on which probes are tried, rather than on a systematic strategy for evidence gathering.
The benchmark itself is open and extensible. The authors have released it with a companion website, and the design supports adding new sandboxes with different rule systems. By providing a testbed where exploration can be measured precisely, ExplorationBench enables systematic comparison of methods for improving how AI systems discover and apply new knowledge — a capability that will become increasingly important as these systems move from answering known questions to investigating unknown ones.