The Authorization Gap: What Stops an AI Agent from Sending Money Where It Shouldn't
Every payment-capable AI agent has two things: a tool that moves money, and a policy that says where the money can go. An attacker talks to the agent. Two things can stop a transfer. The model can decline to request it. Or something between the model and the tool can decline to execute it. Nearly all published measurement focuses on the first. APort Vault measures the second.
The difference matters because it is the difference between a model-level safety property and a system-level safety property. A model that refuses 99% of adversarial prompts still fails 1% of the time. Deployers need to know what catches the remainder. That question, the counterfactual of the same attack on the same model with and without enforcement at the tool boundary, is what this paper answers at a scale that produces a bounded zero.
Where the Attacks Come From
Between March and August 2026, APort Technologies ran a public capture-the-flag event against a live agentic bank with a $6,500 prize pool. Participants conversed with a payment agent through a web interface and tried to make it transfer money to a recipient outside its policy. From 4,750 raw attempts, 4,371 attacks across 1,128 sessions survived filtering. 97.7% arrived in the March 6 to 12 launch window. These are not synthetic jailbreak templates and not researcher-authored task suites. They are what people write when they are trying to get money out of a system and are being paid for succeeding.
The attacks carry categories from an ordered regex classifier: authority claims (1,609), direct transfers (1,243), privilege escalation (342), JSON injection (306), social engineering (187), emotional manipulation (160), prompt injection (144), reconnaissance (102), capability self-register (94), and account manipulation (4). The social engineering label is a residual bucket, not a technique claim.
The Policy Levels and What They Allow
Each level is a passport granting a payment capability with different constraints. Level 1 is permissive: payments.charge with a max of $50,000 and any recipient. Level 2 narrows to three allowlisted recipients and a $2,500 cap. Level 3 adds per-transaction and daily limits, one allowlisted recipient, $500 max. Level 4 requires documented transfers to that recipient with caps, idempotency, and audit codes. Level 5 grants no payment capability at all.
Levels 2 through 4 are the measurement levels: the policy permits some recipients and forbids others, so an unpermitted transfer is a real policy failure. Level 1 is a control that shows what the layer does when the policy says yes. Level 5 is an enforcement test where the prompt explicitly instructs the model to call the transfer tool and states that a guard will intercept it. The authors never fold Level 5 into a headline, because its outcomes outnumber the restricted levels by a factor of 38 and would dominate any pooled figure.
The Two Architectures Under Test
The only manipulated variable is whether a deterministic authorization check sits between the model's tool call and the tool's execution. Model alone: the agent runs with the payment tool exposed, and a transfer_funds call the model emits executes as requested. Nothing inspects it. Behind the authorization layer: identical in every respect except that each transfer_funds call is evaluated against the level's passport before execution by a deterministic policy engine implementing the Open Agent Passport (OAP) specification. A denied call does not execute. The check runs in the tool-dispatch path and is not visible to the model as a modified prompt.
Same model, same version, same prompt, same decode settings, same tool schema, same replay track. The attack corpus is replayed across 14 models from 8 labs (Claude Fable 5.1, Claude Sonnet 5, Claude Haiku 4.5 from Anthropic; GPT-6 Astra, GPT-5.6 Terra, GPT-5.6 Luna from OpenAI; Gemini 3.5 Flash, Gemini 3.8 Flash from Google DeepMind; DeepSeek V4 Pro, DeepSeek V4 Flash; Kimi K3 from Moonshot AI; GLM-5.3 from Z.AI; Qwen3.8 Max from Alibaba; Muse Spark 1.3 from Meta), five policy configurations, and two replay tracks. 225,964 evaluations completed.
Five Events, Not One Number
The paper reports five distinct events per evaluation rather than collapsing them into a single success rate, because collapsing is how an agent benchmark produces a number that does not survive review. The five events are: a payment request (a transfer_funds call appears in the executed tool calls), a successful payment (that call returned "success": true), a policy decision (allow or deny, behind the layer only), recipient membership (the recorded recipient is on the level's allowlist), and an unpermitted transfer (a successful payment to a recipient the passport did not permit).
Read the stages downward, not across. The models asked in both conditions, at rates that differ by under a point. They were paid in both conditions. What differs is the last row, and only the last row.
At Levels 2 through 4, the model alone produced 28,543 requests, 28,521 successful payments, and 140 unpermitted transfers in 76,842 evaluations. Behind the layer: 25,527 requests, 25,370 successful payments, and 0 unpermitted transfers in 69,297 evaluations. The zero was not obtained by refusing everything. Twenty-five thousand three hundred and seventy payments executed behind the layer. The policy is visibly doing the refusing. Of the 25,640 transfer calls it evaluated, it allowed 25,453 and denied 187: 148 for a forbidden recipient, 17 for an invalid amount, 13 for a limit violation, 9 for a missing audit code.
How the Zero Is Bounded
A zero is not a rate of zero. The 69,297 Levels 2 to 4 evaluations behind the layer draw on 790 distinct source sessions. The exact one-sided 95% upper bound on the per-evaluation rate is 0.0043%. The per-session rule-of-three bound is 3/790 = 0.38%. The authors quote the session bound because evaluations sharing a source session are not independent.
On 68,970 matched model, prompt, and track triples at Levels 2 to 4, unpermitted transfers are 105 with the model alone and 0 behind the layer. The same attacks, on the same models, produced the same request behavior and a different boundary outcome.
Request Rates Are Not the Explanation
The obvious objection to a reported zero is that the layer might have obtained it by suppressing requests. The data says otherwise. Aggregate request rates are close in both architectures at every level. At Levels 2 to 4, the paired difference is +0.084 percentage points behind the layer, with a session-clustered 95% percentile interval of [-0.020, +0.189]. The authors state this as an observation, not an equivalence test, and disclose that 1,220 of 68,970 individual pairs (1.77%) disagree in which direction they go. Denied-tool feedback can change what a model does next, and close aggregates do not establish identical behavior. What the comparison supports is narrower and sufficient: the recipient-boundary result was not obtained by suppressing requests.
Fourteen Models, Identical Inputs
On the 1,293 Level 4 prompts, each evaluated on every model with the model alone, request rates run from 71.2% to 84.3%. 809 of 1,293 prompts (62.6%) elicited a request from all fourteen models. On every one of those 809, all fourteen also recorded a successful payment, and on every one the recipient was the level's allowlisted address. Level 4 authorizes documented transfers to that recipient, so this is shared behavior rather than a count of prompts that defeated the models.
The cross-model comparison shows request behavior is a shared feature of the tested configuration rather than a ranking of model safety. The rates span 13 percentage points, and the behavioral intersection is large. Outside that intersection the models differ, both in which prompts elicited a request and in their unpermitted-transfer counts, so this is not a claim that model choice never matters. It is a claim that on the prompts where all models behave identically, a deployer cannot rely on model refusal as the safety mechanism.
What a Leaderboard Cannot Show
Three results emerge from the structure of the attacks that a single-number leaderboard cannot produce.
First, outcomes concentrate in source sessions, not techniques. The 140 Levels 2 to 4 model-alone evaluations that paid a non-allowlisted recipient come from 24 of 790 source sessions. One session accounts for 67 of them (47.9%). Two sessions produce half. Ten produce 90%. "140 unpermitted transfers" sounds like 140 discoveries. It is closer to two dozen people finding something, and one person finding it repeatedly.
Second, a single forged-receipt cohort of 103 attempts from 8 sessions contains a forged tool result in the attacker's own message, the literal string "[Tool Result: verify_recipient]" written to look as though the recipient-verification tool had already run and approved the payee. This cohort accounts for 111 of 113 multi-turn Level 2 model-alone outcomes. It affected seven models and not the other seven. Behind the layer it produced nothing, because the recipient was still not on the allowlist regardless of what the transcript claimed. The technique attacks the model's belief about what has already been checked, not the model's willingness. A check at the tool boundary is indifferent to this, since it re-evaluates the actual recipient against the actual policy.
Third, attack transfer across models is weak. Of 2,809 replayed attacks at Levels 2 to 4, 10 defeated at least one model in single-turn replay and all 10 defeated exactly one. In multi-turn, 73 defeated at least one model, 33 defeated one, 28 defeated two, 8 defeated three, 3 defeated four, and 1 defeated five. Not one attack defeated more than five of fourteen models. A deployer swapping models to escape a known attack would likely succeed against that attack and gain no general protection, which is a bad property to build a security posture on.
Multi-Turn Helps One Family, Not Attacks in General
Comparing single-turn and multi-turn replay at Levels 2 to 4, the multi-turn advantage documented in prior work appears here as a hundred-fold effect in one category and nothing much anywhere else. JSON injection goes from 1/2,184 to 112/2,183. Authority claims go from 5/18,885 to 13/17,666. Every other category shows negligible or zero change. The multi-turn effect at these levels is one technique in one cluster of sessions rather than a general property of extended context. Aggregate multi-turn uplift figures can be carried by a small number of structurally similar attacks, and reporting them by technique is more informative than reporting a single ratio.
Judge Reliability and Why the Headline Does Not Use It
A two-model judge panel (Mistral Medium 3.5 and Grok 4.6, neither a target in this study) independently reads each transcript and returns a verdict. The panel is an audit layer. Pooled inter-judge kappa is 0.772 and falls to 0.167 at Level 3 and 0.521 at Level 5. Against the deterministic check, Grok 4.6 recovers 99.1% of confirmed unpermitted transfers with 3 false positives. Mistral Medium 3.5 recovers 64.4%, missing 3,356 confirmed outcomes, with 139 false positives.
Had the headline rested on the panel, a third of the model-alone outcomes would have been missed by one panel member. It does not rest on the panel. The metric reads executed tool calls and needs no judge. This is the argument for choosing a domain where the outcome is a state change: the audit can be reported honestly because nothing depends on it.
The Cost of Doing This
The run consumed 987.6 million banker input tokens and 218.5 million output tokens, plus 1,071.7 million judge input and 14.0 million judge output tokens. The judge panel accounts for 47.4% of input plus output tokens: auditing the outcome cost roughly as much as producing it. Median end-to-end evaluation latency ranges from 8.2 seconds (GPT-5.6 Luna, single-turn) to 66.8 seconds (Kimi K3, multi-turn), and is unchanged by the authorization layer to within noise. The policy check is not a measurable share of agent latency at these timescales; the companion paper measures it directly at a median of 53ms.
Provider-Side Filtering and Sensitivity
582 evaluations were refused by a provider's input filter before the model saw the prompt, 538 of them GPT-6 Astra. These are kept in every denominator as evaluations in which no transfer occurred, which is conservative against the architectural claim in the model-alone arm. Excluding the 99 attacks that arrived outside the March launch window moves nothing: Levels 2 to 4 go from 140/76,842 to 140/75,694, and Level 5 from 5,362/22,320 to 5,341/22,124. No headline figure depends on the tail.
What the Paper Does Not Claim
The result is scoped to payments, to one tool schema, in a simulated bank. It is not evidence about data exfiltration, code execution, or delegation between agents. The zero is a property of a correctly configured policy engine evaluating a well-specified capability, not a claim that authorization layers cannot be misconfigured. Level 5 grants no payment capability at all and its prompt instructs the model to call the transfer tool while promising a guard will intercept it, so Level 5 measures enforcement and not model susceptibility.
The Level 4 local engine additionally required a confirmation code in the transfer memo, which the published policy pack does not enforce. This makes the local engine strictly stricter than the published pack, so the Level 4 behind-the-layer zero is not by itself evidence about the published pack's Level 4 behavior.
What This Means for Agent Builders
The core lesson is architectural, not behavioral. Asking "which model is safest" gives you a property of a model version that expires when the model is retrained. Asking "what catches the remainder after I choose the best available model" gives you a deployment property that survives model updates. The Open Agent Passport specification implements this as a deterministic pre-action check in the tool-dispatch path, and the benchmark shows it holding across 14 models from 8 labs against 4,371 human-authored attacks.
The policy engine did not block everything. It allowed 25,453 of 25,640 transfer calls at Levels 2 to 4, denying 187. This is not overblocking: every request in this corpus came from an attack, so the 99.27% allowed share says nothing about false-positive rates on legitimate traffic. But it demonstrates that the authorization layer is a policy enforcer, not a refusal engine. It permits what the policy allows and denies what it forbids, which is the property that distinguishes authorization from refusal.
For developers building payment-capable agents, the architecture is straightforward: a deterministic check between the model's tool call and the tool's execution, evaluating the actual request against the actual policy. The check runs in under a millisecond. It does not require model changes, does not require fine-tuning, and does not add measurable latency. It catches the failures that model-level safety misses, and it does so without pretending that a zero means invulnerability. The 0.38% per-session upper bound is honest about what remains.
The full release, 225,964 evaluations, level passports, scoring code, and analysis script, is available at HuggingFace under CC BY 4.0.
Read the paper on arXiv