What Happens When an AI Scientist Works Backward From Data?

Every automated research system starts the same way: someone picks a question. That step, choosing what to investigate, is usually the one humans do before handing the rest to machines. A new open-source project called Urithiru flips that order. It starts with whatever datasets are already sitting around, inspects what columns and variables are measurable, and asks the machine to generate hypotheses from there instead.

Dmitrii spent six weeks building and testing the system, which is now available on GitHub and as a live demo where you can inspect complete search trees, generated analyses, logs, and external validation checks.

Why Start From Data Instead of Questions?

The standard research workflow is question-first: identify a problem, formulate a hypothesis, then go find or collect data. That works well for humans who already have domain expertise. But for an automated agent, the bottleneck is different. Formulating a good question requires taste, and taste requires context that no training dataset fully captures.

The alternative is to start from the measurement surface. Public datasets collected for other experiments already exist in large quantities. NHANES health surveys, economic indicators, climate records, demographic statistics. Instead of inventing new experiments for measurements that do not exist, an agent can inspect what is already measurable and work backward to find questions worth asking.

Monte Carlo Tree Search Applied to Hypotheses

Urithiru borrows its core search mechanism from a familiar place: AlphaGo. Monte Carlo tree search became famous for playing Go, where it explored a game tree by selecting promising moves, evaluating outcomes, and propagating rewards back to parent nodes. The key insight is balancing exploitation of branches that have already performed well against exploration of branches that have not received enough attention.

Urithiru treats each hypothesis as a move and each dataset as a position. The reward is Bayesian surprise: how much testing a hypothesis shifts the model's belief about what the data shows. If an experiment produces a result that contradicts prior expectations, the search engine allocates more budget to that branch. If a hypothesis confirms what was already believed, it gets less attention going forward.

The system also tracks four distinct belief stages for each hypothesis. First, the model's prior expectation before seeing any claim-specific evidence. Second, what a literature search concludes without access to the dataset. Third, what an experiment agent finds after writing and running actual analysis code. Fourth, an optional external validation step using a completely separate dataset.

A Concrete Example: BMI and Blood Sugar

One run explored whether vigorous exercise weakens the relationship between BMI and long-term blood sugar levels. The model's prior belief was moderately confident at 0.77. A literature search lowered that to 0.35, suggesting existing research was skeptical. Then an analysis of 5,141 NHANES participants found the BMI-to-HbA1c slope was 38% lower among vigorously active adults, pushing confidence up to 0.94.

But the external validation step changed everything. A Mexican health survey covering a different population found virtually identical slopes among active and inactive adults, dropping confidence to 0.11. The disagreement between the NHANES result and the external dataset became the reward signal that updated the search tree.

This is exactly the kind of result that makes dataset-first research interesting. The agent found a real statistical pattern in one dataset, then discovered it did not hold up elsewhere. A human researcher would need hours of literature review and data wrangling to reach the same conclusion.

What Did Not Work

The first failed experiment was a persistent memory system. Inspired by Andrej Karpathy's LLM Wiki concept and a strange incident at Hugging Face where an autonomous agent built its own communication protocol, Dmitrii mounted a Markdown wiki into every agent container. The idea was to let agents leave verified observations in a persistent linked structure instead of re-searching the same topics repeatedly.

The wiki lasted two weeks and went through two versions. The first degenerated into one page per hypothesis. The second produced a nicer graph of nine reusable concept pages across 30 discoveries, but it still was not useful. The wiki supplied 46.9 times more context and performed worse than no wiki at all. LLM synthesis smoothed away statistical qualifications and buried relevant evidence under polished prose.

The second experiment was feeding grant objectives into the model. Since dataset-first research has no built-in mechanism for scientific taste, the thought was to let the agent read the grant that funded the dataset and use that as a directional signal. The proposals did shift slightly in embedding space toward the grant language and away from unrelated grants. But executable proposal rates were 269 out of 325 with the grant and 267 out of 325 without it. The grant changed how proposals were phrased, not what the agent could actually do.

External Dataset Replication: Half the Time

The most promising result was the external validation step. The agent took a result from one dataset, searched for a relevant independent dataset, preserved the original claim and variable mapping, and tested the hypothesis again. This worked for nearly half of the frozen claims. Some replicated, some contradicted the original finding, and others stopped because the required measurements did not exist in any available dataset.

This gives researchers a practical intermediate step before collecting new data. Instead of jumping straight to designing a new experiment, you can search existing public datasets, run the same analysis, and treat the result as another data point rather than a final verdict.

Running It Yourself

Urithiru is open-source with Docker and Google Cloud support, checkpoint and resume capability, and a web interface for inspecting the hypothesis tree. The system uses Google's Antigravity agent because it is fast and includes built-in web search, keeping costs low. In testing, the five-hour usage limits were hit after roughly 20 hypotheses.

The project sits in an interesting spot in the broader AI-scientist landscape. Other systems like Sakana's AI Scientist and Google's Co-Scientist focus on end-to-end autonomous discovery. A recent position paper argued these systems are not actually built for autonomous discovery because problem selection requires tacit knowledge that training corpora lack, and preference optimization compresses output diversity toward consensus. Urithiru sidesteps some of these problems by removing the question-selection step entirely and letting the data drive the search.

For developers and researchers interested in automated hypothesis generation, the live demo at urithiru.eamag.me lets you inspect complete runs including the search tree, generated analyses, plots, and external source checks without installing anything.