When you copy a file from your phone to your laptop, open it in a spreadsheet app, tweak a formula, email the result to a colleague, and ask them to check it on their desktop, you are performing a cross-device GUI workflow. It involves six distinct operations across three platforms, with intermediate results passing between them. No existing GUI agent benchmark tests whether an AI can do this. They all evaluate agents on single-device tasks: clicking buttons within one app on one operating system. JarvisGUI, accepted to EMNLP 2026, fills that gap with a benchmark that composes multi-step workflows spanning Android, Windows, and Ubuntu, and evaluates agents on whether they can transfer state, maintain context, and manage long-horizon dependencies across devices.
Why single-device benchmarks give a misleading picture
GUI agents have improved rapidly. Models like UI-TARS, GUI-OWL, and HOLO2 can ground natural language instructions to interface elements with impressive accuracy on individual platforms. Benchmarks such as OS World, OmniBench, and CRAB have tracked this progress, adding dynamic environments and multi-step tasks. But every one of them assumes a single active device at a time. CRAB acknowledges cross-device execution but includes only 18 cross-device tasks, and none of them model realistic file or data transfer between devices.
This limitation matters because real-world workflows are cross-device by default. A developer might pull data from a server terminal, visualize it on a laptop, and share the result through a phone. The intermediate states, file paths, clipboard contents, and login sessions all need to travel with the workflow. An agent that can complete every step within a single app but fails to hand off a file from Ubuntu to Windows is not ready for deployment, even if its single-device scores look strong.
JarvisGUI makes this gap visible. On single-platform atomic tasks, the best model achieves 42.4% success. On multi-device dependent tasks, that same model drops to 6% sub-task success and 2% total task success. The performance cliff is not a marginal degradation. It is a qualitative failure that existing benchmarks completely miss.
A type system for composing workflows
The core technical contribution is a lightweight type system that models GUI tasks as input-output transformations. Each task has typed input slots and output slots. An output slot of type xlsx_file can satisfy an input slot of type ordinary_file because xlsx_file is a subtype. But it cannot satisfy an input slot expecting a url_string. This subtype compatibility rule ensures that when tasks are chained together, the data flowing between them is semantically consistent.
Complex workflows are represented as directed graphs where each node is a concrete task instantiation and each edge represents data flow from one task's output to another's input. The graph structure enforces ordering: if task B depends on the output of task A, B cannot execute until A completes. A global success criterion requires that every task in the graph passes all its evaluators on the final environment state.
The type system enables automated composition. A sampling algorithm selects compatible tasks from a library of 118 atomic templates (24 Android, 56 Ubuntu, 38 Windows), stitches them into a directed graph, and propagates concrete values along edges. For cross-device file transfers, auxiliary transfer tasks move files to temporary directories on the target device via an Alist network drive server. The composed instructions are then rewritten by Qwen3-32B into a single coherent natural language description that preserves all procedural details.
An LLM-as-a-judge stage samples 2,000 composed tasks and evaluates each on three dimensions: realism, coherence, and evaluability, each on a 5-point scale. Only tasks scoring maximum on all three are retained. Manual inspection of 50 sampled tasks found 92% fully correct, with a Cohen's Kappa of 0.92 between LLM filtering and human judgment, confirming strong agreement.
Three platforms, 442 subtasks, one evaluation framework
The benchmark spans three operating systems running in Docker containers: Android (via Docker-based emulation), Windows (via KVM-enabled virtualization), and Ubuntu (native Docker). Each platform exposes a unified observation space of screenshots and a unified action space that masks OS-specific differences: click, type, swipe, key press, scroll, with platform-appropriate variants (e.g., left_click vs. long_press on Android).
JarvisGUI contains 150 compositional tasks evenly split into three categories: single-platform with dependencies (SW), cross-platform without dependencies (MI), and cross-platform with dependencies (MD). These decompose into 442 platform-specific subtasks: 187 Ubuntu, 138 Windows, 52 Android, and 65 file transfer assistant tasks. The evaluation is programmatic: each task has parameterized Python evaluators that inspect observable artifacts (file content, DOM elements, clipboard state) on the final environment state, ensuring robustness to stochastic interface changes.
Current agents fail hard on cross-device workflows
The paper evaluates six open-source GUI agents using a Planner-Grounder architecture. Qwen3-VL-Plus serves as the centralized planner, receiving screenshots from all three platforms and the user instruction. It follows an Observation-Planning-Action protocol to select the target platform and abstract action, without producing coordinates. Each baseline model (UI-Venus, UI-TARS-1.5, MAI-UI, GUI-OWL, Qwen3-VL-Instruct, HOLO2) serves as the grounding agent, predicting exact click coordinates on the selected platform's screenshot.
On atomic tasks, HOLO2 leads at 42.4% overall success, with GUI-OWL at 39.0% and UI-TARS-1.5 at 37.3%. Performance is strongest on Ubuntu (up to 60.7% for HOLO2) and weakest on Windows (7.9% to 18.4%), suggesting a pre-training data distribution bias toward mobile and command-line interfaces.
The compositional results tell a different story. On multi-device dependent tasks (MD), the best total task success rate is 8% (GUI-OWL). Sub-task success rates for MD range from 1.3% to 8.0%. On single-device dependent tasks (SW), total task success rates range from 4% to 16%, and sub-task success rates are at or near zero for four of six models. Multi-device independent tasks (MI) fare better, with total task success between 23.2% and 28.8%, but still significantly below atomic performance.
The pattern is consistent across models: cross-device dependencies create fragile critical paths. A single intermediate failure, such as an inability to locate a file or a login timeout during cloud transfer, breaks the entire chain. The agent then treats unexecuted dependency steps as completed, producing cascading errors. Task success rate drops to near zero for workflows requiring four or more subtasks, regardless of model.
Four failure patterns that cut across models
The error analysis identifies recurring system-level failure modes. First, premature termination: the agent declares the task complete before all instruction requirements are met, often after successfully executing the first two or three steps of a five-step workflow. Second, omission of essential intermediate actions: the agent skips steps that produce inputs needed by later steps, then attempts those later steps anyway, failing because the expected state does not exist. Third, incorrect sequencing: the agent performs actions in the wrong order, such as trying to open a file before it has been transferred. Fourth, platform routing errors: the agent fails to map implicit instructions to the correct platform, interpreting "back to the computer" as a different device than the one previously used, or not recognizing that "C drive" refers to Windows.
HOLO2 trajectory analysis breaks cross-device failures into three categories: failures originating in the underlying single-device subtasks, failures in the cross-device transfer mechanism itself, and failures caused by loss of context between tasks. The third category is particularly insidious because the agent loses track of what it was doing mid-workflow, even when individual subtasks would be executable in isolation.
What this means for building real-world agents
The performance gap between atomic and compositional tasks is not a tuning problem. It reflects missing capabilities in current agent architectures. The Planner-Grounder design processes one platform per step, receiving a single screenshot and producing a single action. But cross-device workflows require reasoning about the state of devices the agent is not currently observing. The agent needs to infer that a file transferred from Ubuntu should now be accessible on Windows, that a clipboard value from the phone should be pasted into a desktop application, that a login session on one device grants access to resources on another.
This inference requires maintaining a mental model of the entire workflow state, not just the current screen. The type system and task graph structure in JarvisGUI formalize exactly this: each task's output slots become the next task's input slots, and the graph encoding ensures that downstream tasks receive concrete values from upstream results. An agent architecture that internalizes this structure, rather than treating each step as an independent decision, would likely perform substantially better.
The benchmark also reveals that the composition pipeline itself is reusable. The type system is not specific to any single application domain. Any GUI task that produces typed outputs (files, URLs, text strings, structured data) and consumes typed inputs can be slotted into the framework. This makes JarvisGUI extensible: adding new atomic tasks to the template library automatically enables new compositional workflows without manual annotation of the full cross-device task set.
For developers building GUI agents, the practical implication is clear: single-device benchmarks are necessary but not sufficient. An agent that scores well on OS World or OmniBench may still fail catastrophically when asked to coordinate across platforms. The 150 compositional tasks in JarvisGUI, spanning 29 apps across three operating systems, provide a concrete test suite for the capabilities that matter in deployment: state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management. These are the capabilities that separate a demo from a product.