Computer-use agents have made real progress on benchmarks like OSWorld and AndroidWorld, but they still spend most of their time clicking through GUIs. A task that could be completed with a single command line operation becomes a long sequence of visual element detection, coordinate clicking, and state verification. Real computer work is hybrid: people use the GUI to inspect visual state and navigate interfaces, then switch to the command line for batch operations, precise transformations, and high-throughput processing. The gap between how agents actually work and how people work is an efficiency and capability problem that existing benchmarks do not capture.
Why Hybrid Environments Are Hard to Build
Building an environment where an agent can use both the GUI and CLI over the same application state requires two things: reproducible application setup and usable command-line surfaces. The first is straightforward but tedious: each application needs installation, configuration, and dependency management. The second is the real bottleneck. Desktop applications expose different automation surfaces. Some have native command-line tools (Blender has blender --python-expr, VLC has cvlc). Others expose scripting APIs (GIMP has Script-Fu, LibreOffice has UNO). Many have no usable CLI at all, requiring either reverse-engineering internal protocols or building custom tooling from scratch.
Existing approaches handle this in one of two ways. GUI-centric environments like OSWorld provide screenshot-based interaction but no CLI access, so agents must click through everything. CLI-centric environments like TerminalWorld provide command-line access but no visual state, so agents cannot handle tasks that depend on layout or interface appearance. Neither provides the shared application state that makes hybrid interaction possible.
The CUA-Universe Pipeline
CUA-Universe is an environment-to-data pipeline with three components that turn real desktop software into hybrid GUI+CLI environments and generate training data from them.
App-Forge handles application adaptation and tool construction. An installer agent operates a VM harness to install and configure each application, diagnosing failures interactively until successful launch is verified. The setup is distilled into a reproducible configuration. For tool construction, the system draws from three sources: native command-line tools it discovers, scripting APIs it wraps, and agent-native CLIs it generates when existing interfaces are insufficient. The generator is inspired by CLI-Anything, creating custom command-line surfaces that expose application functionality through a unified interface. GUI and CLI operate over the same project state, with lightweight adapters resynchronizing stale GUI views after external CLI edits. This scales to 16 desktop applications across diverse domains: creative tools (Blender, GIMP, Godot), office software (LibreOffice Writer, Calc, Impress), media players (VLC, Audacity), technical tools (QGIS, VS Code), and web browsers (Chrome).
Task-Weave synthesizes hybrid tasks of controllable difficulty. First, it builds a reusable operation pool by running exploration agents with diverse goals across seed projects, then sliding a window over trajectories and prompting an LLM to abstract interactions into high-level reusable operations (like export_scene_to_gltf rather than click). Operations are deduplicated, clustered, and aggregated into a global pool. Then, task instantiation composes these operations into diverse tasks conditioned on real seed projects (.blend scenes, .odp decks). Difficulty is controlled by chain length and composition. Finally, a ReAct-style review agent grounds each task in real execution, checking feasibility, unambiguity, and whether the seed already satisfies the goal. Invalid tasks are discarded.
Path-Steer converts tasks into efficient training trajectories. A hybrid execution interface lets agents emit either GUI actions or CLI actions at each step. CLI actions are parsed against the application's tool registry, expanded into concrete commands, and executed in the VM. An efficient-path steering mechanism derives a hybrid execution prior from each task's operation chain, indicating when an operation is better suited to the CLI (batch, precise, high-throughput) or the GUI (visual layout, interface state). This provides modality-level guidance without specifying low-level actions. Completed rollouts are scored by a VLM judge, and high-scoring hybrid trajectories are retained for post-training.
Results: A 9B Model Matches Closed-Source Performance
The training pipeline generated 4,923 verified episodes with approximately 235,000 step-level records across 16 applications. The authors fine-tuned Qwen3.5-9B with LoRA on these trajectories for 3 epochs on 8 A100 GPUs in roughly two days.
On CUA-Verse, a held-out benchmark of 160 hybrid tasks across eight desktop applications, the trained model achieves 0.582 score, the best among open-source models. It surpasses Kimi K2.5 (0.522) and trails only the proprietary Seed2.1 Pro (0.599) and GPT-5.5 (0.768). Against the identical base Qwen3.5-9B (0.189 score), it achieves roughly 3x improvement with 37% fewer steps and 60% fewer tokens (255K vs 643K per episode).
On OSWorld, the model gains +16.8 success-rate points over GUI-only execution, converting 41 previously failed tasks into successes. This is by far the largest net gain of any agent: GPT-5.5 gains +3, Seed2.1 gains +8, and Kimi K2.5 is essentially flat (+2). The CLI interface is where the gain concentrates. On jointly solved tasks, the CLI interface uses 2.35x fewer steps and 1.79x fewer tokens.
On OSWorld-MCP, which augments OSWorld with 158 MCP tools and a different tool-invocation interface never seen during training, the model improves score from 20.90% to 28.69% (+7.79 points) and more than doubles tool-use accuracy from 10.66% to 23.36%. A single 9B model surpasses Seed2.1 Pro on score (29.51% vs 27.03%) and approaches GPT-5.5 and Kimi K2.5.
What the Numbers Reveal About Hybrid Interaction
The efficiency gains are the most telling result. Adding CLI access to a model that was not trained to use it yields marginal improvements: GPT-5.5 gains +3 tasks, Seed2.1 gains +8. But training a model specifically on hybrid trajectories produces a qualitatively different interaction pattern. The trained model does not simply add CLI commands to its repertoire. It learns when to switch modalities: use the GUI to locate targets and inspect visual state, then use the CLI to process them in batch or apply precise transformations.
The structured nature of the capability is visible in the per-application breakdown. The model is strongest on audio/video applications (Audacity 0.815, OBS 0.605) where CLI operations like audio filtering and video encoding are natural, and weaker on 3D/spatial applications (Blender 0.398, Godot 0.460) where visual layout and spatial reasoning dominate. This is the kind of modality-dependent performance profile you would expect from a model that has learned to match interface to task, not one that applies a uniform strategy everywhere.
Why This Matters Beyond the Benchmark
The transfer results are the strongest evidence that the learned capability is general. The model was trained on application-specific CLI tools in CUA-Universe and never saw the MCP action space, yet it generalizes to OSWorld-MCP with significant gains. This suggests that the pipeline teaches a transferable skill: how to reason about when GUI and CLI are appropriate, not just how to use specific tools. The orchestration pattern learned in one context applies to a different tool interface in a different benchmark.
For practitioners building computer-use agents, the immediate lesson is that hybrid interaction requires hybrid training. Exposing a CLI to a GUI-only agent does not make it a hybrid agent. The agent needs to learn, through experience on tasks that require both modalities, when each interface is appropriate. CUA-Universe provides a scalable way to generate that experience: turn real applications into hybrid environments, synthesize tasks that require both GUI and CLI, and harvest trajectories that demonstrate effective orchestration.
The framework scales through agent-driven construction rather than per-application manual engineering. App-Forge reuses the same adaptation workflow across applications. Task-Weave composes operations from a reusable pool. Path-Steer harvests trajectories from efficient rollouts. Each environment becomes a continuous source of tasks and training data, not a one-time benchmark.
CUA-Universe will release its code and data. The pipeline covers 16 applications with tool inventories ranging from 10 commands (Thunderbird) to 46 commands (GIMP). The training data spans creative, office, media, and technical domains, providing the diversity needed for generalizable hybrid interaction skills.