Scaling capital access across agent runs

FrontierML enters the landscape as a benchmark from Induction Labs that measures how effectively agents convert budget into model performance on end-to-end machine learning tasks. The project addresses a question growing in relevance as AI agents take on more research work: when given increased funding, can they produce substantially better models? The benchmark scales spending rather than just reasoning depth, giving agents $10, $50, and $200 per task to spend on training compute and data, then evaluating whether additional capital is used productively.

Task design without prescribed solutions

Each FrontierML task defines a problem without prescribing a solution, leaving the entire development process open-ended. Agents decide what model architecture to build, what data to use, which GPUs to rent and for how long, which experiments to run, and how to serve the result. The benchmark contains 21 long-horizon tasks inspired by real work at frontier labs, neolabs, and academic institutions. Tasks range from converting an autoregressive LLM into a text diffusion model to building a streaming end-of-turn detector for voice assistants to writing a bot that plays competitive Pokémon battles. Because agents control the full pipeline, they can turn additional capital into better performance through larger training runs, better data, or more algorithmic experimentation.

Verification and reward hacking prevention

To address the large reward hacking surface of open-ended long-horizon tasks, FrontierML employs several verification strategies. Agents must submit a standalone package of serving code and model weights that runs with no internet access on a single H100 machine. Each task provides limited validation samples and evaluates on a large out-of-distribution held-out test set not shown to the agent during development. For example, in the Ternary PTQ task only two of four evaluation benchmarks are disclosed publicly (GPQA-Diamond, LCB v6) while the held-out set includes MMLU-Pro and MATH-500. An LLM judge review agent examines each submission and the agent’s trajectory for violations including searching for answers, training on test splits, and invalid model provenance. Strict time limits bound the allowed inference compute; if the submitted server does not return answers for each held-out sample within the limit, the agent scores zero on the task.

Twenty-one real-world ML tasks

The 21 tasks span language, vision, audio, and games, drawing inspiration from research at frontier labs, neolabs, and academic institutions. This diversity ensures agents must adapt across modalities rather than specialize on a single problem type. The open task interface allows agents to start from any public model and architecture, just as a human researcher could, creating a more faithful evaluation of end-to-end model training capability.

Evaluation setup and constraints

Each task provides the agent with identical resources: a 12-hour CPU sandbox, a wallet containing $10, $50, or $200, a few validation samples, and a task description. Several constraints ensure tasks are not trivially solvable, such as restricting use of already-trained public specialist models or training on the test split of any public benchmark. The minimal information provided requires agents to make decisions about architecture, data, and compute allocation from the outset. Before submitting, agents can test their work with frontier selfcheck, which runs the current submission against public validation samples up to four times per trial. Evaluating an agent on all 21 tasks costs up to $5,460 in wallet spend, not including model API costs.

Results from the leaderboard

The leaderboard ranks 14 top-performing models based on their final scores, wallet spend, and model cost. Claude Opus 5.5 achieves the highest score at 58.8 with $127.67 wallet spend and $55.06 model cost. Claude Sonnet 5.5 follows with 50.4 points from $125.17 wallet spend and $46.95 model cost. GPT-6 Astra scores 44.5 with $100.40 wallet spend and $116.22 model cost. Claude Fable 5.1 reaches 41.9 with $79.02 wallet spend and $102.84 model cost. GPT-6.1 Sol scores 38.7 with $103.40 wallet spend and $10.96 model cost. Claude Haiku 5.5 achieves 34.9 from $58.92 wallet spend and $10.38 model cost. Kimi K3 scores 20.3 with $61.36 wallet spend and $48.79 model cost. GLM 5.3 reaches 19.6 from $41.58 wallet spend and $27.62 model cost. Muse Wonder scores 19.3 with $33.39 wallet spend and $17.06 model cost. Qwen3.8 Max achieves 17.5 from $65.43 wallet spend and $18.54 model cost. Muse Spark 1.3 scores 16.0 from $41.22 wallet spend and $17.71 model cost. GPT-6 Luna reaches 13.1 from $17.15 wallet spend and $0.75 model cost. Grok 4.7 scores 10.9 from $20.22 wallet spend and $30.02 model cost. Gemini 3.8 Flash achieves 5.7 from $2.78 wallet spend and $9.75 model cost.

Architecture innovation during evaluation

Because FrontierML keeps tasks open-ended, agents design their own architectures and produce their own data. In 28% of trajectories, agents add newly trained parts to pretrained models or train a new model from scratch. Of the 735 runs that trained a model, 14% used only data generated with an LLM, a local model, or a renderer or simulator, 44% combined public datasets with generated data, and 35% used public datasets only. Seven percent trained on the task's own data alone. Agents almost always write their own training loop: 87% of training runs use a hand-written PyTorch loop, and 8% use the Hugging Face Trainer. Twenty-six percent of runs train a LoRA through the PEFT library.

Budget spending patterns in practice

On average, agents only spend 46% of the capital in their wallet, even when explicitly told there is no penalty for using their entire wallet. No individual agent uses more than 80% of the capital available at any budget. While FrontierML provides agents with a maximum time limit of 12 hours, agents often submit much earlier. On average, frontier models only spent half of the available time, while no model spent more than nine hours on average.

Inefficiencies in agent spending

Agents rarely researched their options before spending money, instead relying on prior knowledge to choose models for synthetic data generation and GPUs to rent. Synthetic data generation: agents almost always chose models from memory without searching for newer models beyond their knowledge cutoff. Only 3 runs listed the set of available models on OpenRouter before generation. Over 67% of all calls to OpenRouter for synthetic data were to models more than one year old. GPU selection: across all agents, the vast majority of compute was in H100s, with 60% being spent on H100 SXM GPUs. Over all runs, only two web searches were performed about which GPU to use, both from Grok 4.7.

Comparing FrontierML to existing benchmarks

FrontierML more faithfully measures whether agents can autonomously train other models end-to-end while handling the complexity a human research team would apply. Existing benchmarks are narrower in scope and give up task diversity or realism. Terminal-Bench 4.0 limits tasks to small parts of the model training stack, with only three of 66 tasks giving the agent a GPU. MLS-Bench focuses exclusively on machine learning science, improving one component of a fixed codebase on fixed hardware. PostTrainBench covers only LLM post-training with fixed base model and hardware, prohibiting use of external APIs for distillation. MLE-bench uses Kaggle puzzles from 2012 to 2024 rather than modern real-world tasks, with fixed hardware and no mechanism to spend money. Autoresearch Bench and AutoResearchExam give models no internet access and tasks are not open-ended, focusing on improving small components of existing systems. PostTrainBench is the closest prior work, but it does not give models freedom to design architectures or generate synthetic data like FrontierML does, reducing model separation.

Future directions and current limits

Current evaluation runs use budgets of $10, $50, and $200, but some machine learning tasks are intractable at these budget levels, limiting the available task set. The FrontierML team is working on tasks with larger budgets ranging from $1,000 to $100,000, and on tasks requiring multi-node training and significantly more complex infrastructure. Each agent was only run at one reasoning effort; future releases plan evaluations across multiple reasoning efforts to provide a more granular Pareto frontier across total spend. Agents are currently limited to on-demand AI neoclouds and OpenRouter; planned expansions include access to more paid services such as Tinker and human talent marketplaces.