Banks Built a Benchmark That Exposes How Far AI Agents Are From Professional Grade
A team of bankers from Morgan Stanley, Bank of America, UBS, and Rothschild and Co spent months constructing a testing environment that mirrors how investment banking actually works. Their benchmark, called ATLAS-Finance, contains 100 expert-level tasks spread across 13 detailed firm environments. The results, drawn from 11 frontier models, show that even the best systems pass barely one in eight tasks.
The benchmark covers nine industries and more than 20 financial-analysis methodologies. Each task would take a human analyst 15 to 30 hours to finish manually. The environments include full inboxes, shared drives, notebooks, calendars, and dozens of coworkers and clients who send updates mid-task. The goal was to test agents under conditions that reflect real working life rather than clean, pre-packaged prompts.
From Workflows to Workplaces
Previous benchmarks moved from short-form question answering to end-to-end financial workflows. BankerToolBench, for instance, tested agents completing self-contained junior investment banker tasks. Those benchmarks showed that frontier agents can execute parts of a workflow but remain unreliable at the standard required for professional delegation. Their environments were simplified and static, which excluded the scattered, evolving, and sometimes contradictory information that real analysts navigate daily.
ATLAS-Finance shifts the unit of evaluation from a linear workflow to a workplace world. Each environment contains an advisory firm with specialized teams, senior and junior professionals, clients and counterparties, active and background mandates, communication histories, shared files, permissions, and firm-specific conventions. Before the agent populates a single spreadsheet cell, it must assemble its workflow details from environment context, understand its relationship to each counterparty, and resolve controlled contradictions by identifying the right decision-maker. Some tasks include real-time updates that correct prior assumptions, requiring the agent to incorporate changes before delivery.
Grading Like a Senior Banker
Every task carries between 42 and 486 binary rubric criteria. In total, 14,600 environment-grounded criteria span the benchmark, averaging 146 per task. All criteria were authored and verified by industry experts. Each criterion carries a weight reflecting its relative importance.
The grading mirrors how a professional would audit work. Schedules contain gating dependencies. If the debt schedule lacks live formulas, everything downstream is marked unreachable, not merely wrong. A run passes only when it satisfies every criterion weighted three or higher and triggers no penalties. Penalties catch fabricated data or a missing audit trail.
The system that performs the grading is Gandalf the Grader, an agent-as-judge framework designed to verify financial artifacts. Beyond a score, each failing run produces a diagnosis: which schedule broke, what the break cost downstream, and whether the model lacked the knowledge, applied it incorrectly, or built correctly but delivered static values instead of working formulas.
What Eleven Models Revealed
All 11 models were run within the OpenCode agentic harness. Claude Opus 5 achieved the highest pass rate at 12.3%. Claude Fable 5.1 reached 12.0%, and GPT-6 Astra scored 11.3%. The remaining eight models all fell below 10%. A pass requires meeting every critical rubric criterion and triggering no penalties, which sets a demanding bar.
The researchers analyzed 1,100 rollouts across models to understand where agents fail. They traced each failing criterion through the submitted workbook, governing inputs, and dependency chain to its earliest repairable root cause. Financial logic and methodology issues dominated, accounting for 52.2% of all failures. Adding finance integration issues brought the total to 53.7%. Scope and completeness problems made up 26.6%. Spreadsheet engineering accounted for 10.8%. Presentation and delivery errors represented 8.8%. Source and data handling issues were negligible at 0.1%.
The recurring pattern is that models can execute individual steps competently but cannot reliably apply correct financial methodology under realistic conditions. The mistakes they make are plausible-looking and appear across all model families.
The Project Osprey Breakdown
The benchmark documentation includes a detailed walkthrough of one task that illustrates the failure pattern. Project Osprey is a full leveraged buyout rebuild involving a 338(h)(10) tax election for a payments target acquired in 2019. The task begins with a staffing kickoff, exactly how a junior banker receives a new assignment. The agent gets a heads-up message pointing to a project details email. The delivery contract specifies an exact filename, recipients, subject line, and the requirement that the workbook contain live in-cell formulas.
A Claude Opus 5 agent completed the task in 73 execution steps. It searched the assignment subject, read the thread and seven related threads, found the desk's OneNote with modeling approach notes and call notes, listed and downloaded every attachment, and programmatically dumped every source workbook to read cell values. It then wrote the complete LBO model in Python before building the workbook. All thirteen integrity checks on the control sheet read OK. The workbook contained 2,641 live formulas and zero error cells.
The rubric score was 0.336 out of 1,460.
The failure traced to a single decision during discovery. An email from the client's tax lead listed each input for the tax-basis build with its explicit operation: "added to purchase price," "subtracted in the basis calc," "subtracted." One line, Existing Goodwill, carried no operation because under the record's method it plays no role in the amortizable step-up. The agent's Python shadow model contained a silent line that subtracted existing goodwill from the amortizable basis. No comment flagged the decision, no alternative treatment was considered, and nothing was marked for review. Six steps later, the agent rewrote the label itself as "Existing Goodwill carry-over tax basis," recording an unassigned treatment as source data.
From that point, every self-check verified against the shadow model, so all integrity checks read green. But the subtraction cut every year's tax amortization by roughly a quarter and understated the election's value. The delivery email walked the client through the netted figure as if it were the record's method. Because the rubric records dependencies professionally, once the tax construction was wrong, all 180 points in that section were marked unreachable. The purchase-price bridge lost points similarly, and revenue drift propagated through all 530 Returns points.
Every Claude Opus 5 run on this task makes the same deduction and lands on the same invented figure. Kimi K3 and GPT-5.6 Sol arrive at the identical incorrect treatment independently. The same error appears across different model families, suggesting it is a structural weakness rather than a quirk of any single system.
What This Means for Deploying AI in Finance
The benchmark's designers make a practical argument. If an agent makes a critical error in a client deliverable, senior staff must audit all of its work. The time savings and expertise acceleration that justified the investment disappear. In competitive industries, teams have lost mandates and millions of dollars in revenue over smaller errors than the ones documented here.
ATLAS-Finance is not the final word on AI capability in finance. It is a testing framework that surfaces where the gap between current models and professional standards remains widest. The benchmark is open, and the researchers invite the community to use it, improve it, and build on it. Until agents can consistently pass these tasks, the practical deployment of AI in roles that require professional-grade financial judgment remains limited.