Text-to-SQL benchmarks have become the default way to measure progress on data agents. Spider, BIRD, and their variants ask a model to write a query, then check whether the result matches a gold answer. The best models now clear 90 percent on these leaderboards. But audits have shown the gold answers themselves are frequently wrong, and the underlying datasets bear little resemblance to an enterprise warehouse. A business event in a real ERP system touches dozens of tables across finance, logistics, and HR. In the public benchmarks, that same event fits in a single table. The gap between benchmark performance and production reality has grown too wide to ignore.

Why existing benchmarks fail enterprise workflows

The core problem is data sensitivity. Real enterprise warehouses contain payroll, customer PII, financial ledgers, and competitive intelligence. No company releases them. Benchmark creators therefore cobble together public datasets — CSV dumps, Kaggle competitions, synthetic generators — where referential integrity is optional and a "complex" query joins three tables. The resulting tasks test SQL syntax, not the ability to navigate a 200-table schema where the same entity appears under different keys in different modules, where slowly changing dimensions require point-in-time lookups, and where the answer key doesn't exist because the question is "should we ban this courier?" not "what is the count?"

Argo-Bench addresses this by simulating a complete business instead of sampling tables. The authors constructed a food-delivery platform operating in New York City at 2024 scale: 81 million orders, 3.4 million customers, 180 thousand couriers, 24 thousand merchants. The simulation draws on public data — NYC Department of Consumer and Worker Protection reports, 10-K filings from public companies, academic studies on gig-work economics — and calibrates per-delivery economics to within 5 percent of regulatory figures in 13 of 16 quarterly comparisons. Fraud patterns are seeded from documented cases: GPS spoofing, order theft after pickup, bot-driven offer grabbing, account renting, promotion farming, stolen-card account takeovers, storefront refund collusion. Each pattern is tuned to real-world separability and base rates so that detector precision reflects honest device, address, and card sharing.

The warehouse: 235 tables, 7.5 billion rows, Oracle E-Business Suite schema

The simulated world exports to an analytics warehouse modeled on Oracle E-Business Suite 12.2, a widely deployed ERP whose data dictionary is publicly documented. Three ERP consultants with 14 to 31 years of experience designed the schema. Every standard table and column exists in Oracle's Vision demo instance. Unused flexfields and modules — shipping, inventory, foreign currency, withholding tax — are omitted because they would be empty for this business. The result is 235 tables and 7.5 billion rows. Custom extensions alone account for roughly a third of tables and rows.

Crucially, the warehouse avoids the drift and inconsistency that accumulate in real systems over years of migrations. There are no deprecated tables overlapping active ones, no figures that fail to reconcile across modules. This is a deliberate choice: if legacy and current tables disagreed, the correct answer would depend on undocumented tribal knowledge. Instead, the warehouse reflects a production workload that reuses only a few dozen combinations of the hundreds of tables, forcing an agent to discover which ones matter by exploring.

The simulator's ground-truth state — the actual fraud labels, the true courier earnings, the real order timestamps — is withheld. The agent sees only the warehouse export. To answer any question, it must reconstruct facts by joining across modules, resolving key mismatches, and handling slowly changing dimensions exactly as a human analyst would.

Task design: 210 workflows across five business domains

Argo-Bench comprises 210 tasks grouped into five areas. Trust and Safety tasks detect and act on fraud: ban courier accounts, hold merchant payouts, flag customer rings. FP&A tasks forecast unit economics and rebuild finance dashboards. Marketplace tasks forecast courier supply and allocate incentive budgets across zones and shifts. Accounting tasks report final values after month-end close. Growth tasks measure promotion effectiveness and publish membership metrics.

Each task has four components: a prompt written as a stakeholder would ask it (loose, high-level, no template), a scope defining the last visible month of 2024, a list of expected filings with their grading modes, and a frozen answer key derived from the simulator's latent state or SQL over latent tables. Prompts average 159 words. Fifty-one tasks require more than one filing. The reference solution for one task joins six tables across three EBS modules to recover the minimum-pay rule change and fit a prediction interval.

Actions, not queries: the mission-control interface

The defining departure from text-to-SQL is what the agent produces. Instead of a SELECT statement, the agent files actions through a Python library in its sandbox: ban these accounts, allocate this budget across these zones, issue this back-pay amount, publish this dashboard data source. The grader scores each filing by its consequences in the simulator.

A ban list is scored by fraud losses prevented — including fraud the platform never detected — net of revenue lost from wrongly banned customers. A forecast is scored by weighted interval score against a reference forecast fixed before the outcome, so filing the true median and interval is the optimal strategy. A budget allocation is scored by the share of attainable savings the simulator realizes. A dashboard data source is scored on structural contract compliance and numerical accuracy. This design permits tasks requiring ML, operations research, and optimization libraries that SQL cannot express. It supports end-to-end workflows: emitting correct SQL is not enough when the task demands rebalancing courier incentives, banning likely fraudsters, and holding suspicious payouts. Filings are explicit declarations of intent, making partial credit well-defined. Grading is objective — no LLM-as-judge.

Of the 210 tasks, 178 require account actions, forecasts, budgets, or dashboards. Ninety-nine tasks are cutoff at a specific month so forecasts are graded on unseen future periods. Every task has an executable reference solution demonstrating solvability using only the warehouse.

Evaluation: 14 models, sobering results

The authors evaluated 14 frontier and open-weight models using Inspect AI on Google Kubernetes Engine. Each agent runs in a gVisor sandbox with 25 preinstalled Python libraries, no internet, and a 500-turn limit. Models run at default sampling; open-weight models via Fireworks endpoints, proprietary via developer APIs.

The strongest model, Claude Opus 5.5, scores 95 or higher on only 34.8 percent of tasks and averages 59.5 points. Nine of the 14 models average below 35. By domain, Opus leads in Fraud, Finance, and Compliance. GPT-6 Astra leads in Forecasting. Claude Sonnet 5.5 leads in Compliance. More reasoning effort helps GPT-6 and Claude models at every step, with diminishing returns: Opus gains 17.0 points from low to medium effort, 5.9 from medium to high, 4.0 from high to extra-high. Sonnet gains 14.9 and 11.5 over the last two steps. Gemini 3.8 Flash gains 8.6 from low to medium. Qwen 3.8 Max gains 5.2. Muse Spark 1.3 and DeepSeek V4.1 Flash barely move past medium.

Cost and steps vary widely. Opus 5.5 uses 82 steps and $4.71 per task. GPT-6 Astra uses 85.7 steps and $2.71. Proprietary models range from $0.06 (Claude Haiku 4.5) to $3.94 (Gemini 3.8 Flash). Open-weight models range from $0.35 (GLM 5.3 Flash) to $4.89 (Muse Spark 1.3), reflecting higher call counts.

Where models fail: wrong record, wrong objective, wrong quantity

The paper highlights failure modes that are instructive for practitioners. Many failing runs use sound methods but read the wrong record, optimize the wrong objective, or measure the wrong quantity. In a task sizing the courier location service, 46 of 47 completed runs miss all 12 monthly counts despite a 1 percent tolerance. The warehouse keeps hourly idle check-ins; the app sends them every half hour. Only Sonnet 5.5 at extra-high effort reconstructs the missing half-hourly check-ins. Across dashboard tasks, 66.9 percent of data sources that pass structural validation score zero on values. Across 4,553 forecast series from 3,346 runs on 72 tasks, nominal 80 percent intervals contain the realized value only 44.8 percent of the time.

Passing runs share a habit: they check definitions against a second source. They verify that "active courier" means what they think it means before building a forecast on it. This suggests many failures are preventable with better exploration discipline.

Limitations the authors acknowledge

The dataset differs from the most complex ERP deployments in four ways. First, it covers a single city — no foreign currency, no transaction/local/reporting currency representations. Second, many warehouses combine multiple businesses (food delivery, grocery, rides) with shared accounts like driver payables and customer credits mixed in one balance. Third, only one year is simulated; a longer history would add 2020–2022 market shocks to operations and forecasts. Fourth, only Oracle EBS format is supported; SAP S/4HANA would be a natural extension.

The simulator validates 23 distinct metrics, but some in-world behaviors rest on weakly grounded assumptions. The membership program in particular lacks strong evidence. A simulator encodes its authors' assumptions, and a generator sharing the simplifying assumptions of systems under evaluation can make tasks easier than reality. Calibration to aggregate targets does not guarantee realistic tails. Argo-Bench compares agents; it does not estimate performance on a real company's warehouse.

What this means for practitioners building data agents

Argo-Bench offers a concrete snapshot of the field. The strongest model solves roughly one-third of enterprise-scale tasks. The best open-weight model solves 14.3 percent. The gap is not from lack of effort — more reasoning helps, but with diminishing returns — nor from a single methodological flaw. Models analyze the wrong quantity, optimize the wrong objective, read the wrong record. The remedy appears to be exploration discipline: verify definitions, cross-check sources, understand the warehouse before acting.

The public release includes one world's warehouse on Hugging Face (CC BY 4.0), tasks, reference agent, tool server, and sandboxes (Apache 2.0). A demo at argo-bench.com lets you browse the released world at 10 percent scale. Leaderboard scores are computed on a second, private world, so memorization cannot game the benchmark.

The broader lesson: query-matching benchmarks measure something too narrow to predict enterprise data work. Argo-Bench's design — actions graded by consequences in a simulated warehouse — points toward evaluation that reflects what analysts actually do. The next step is building agents that can guide, reason over, and act within enterprise-scale data environments, not just generate SQL against small, well-documented schemas.

Read the paper on arXiv