Why screenshot-based automation fails
Current computer-use agents capture the screen, send the image to a multimodal model, and ask it to return x,y coordinates for clicks. Each look costs 1,000 to 1,800 tokens. The model guesses coordinates from pixels, then another screenshot confirms whether the click landed. A single Calculator task on Windows 11 measured in October 2026 required 2 to 8 screenshots, consuming 2,072 to 8,288 tokens at 1024×768 resolution. Coordinate drift from DPI changes, window resizing, or layout shifts breaks these workflows silently.
How the a11y tree replaces pixels
clawdcursor queries the native accessibility API first. On Windows it uses UI Automation, on macOS it uses NSAccessibility, on Linux it uses AT-SPI over DBus. The tool builds a confidence-scored element map called el_NN that fuses tree attributes with OCR results when the tree is sparse. The model receives text it can act on by name: 38 tokens to find a button, 50 to read a field. No image bytes reach the model unless the application is canvas-only or requires spatial reasoning. Vision stays a last resort.
Actions target stable element IDs. A button labeled "Equals" gets invoked by name. Coordinates serve only as a fallback. The accessibility helper lists 20 windows in 0.3 seconds and reads a single window tree in 0.06 to 0.2 seconds. This speed keeps the agent responsive during interactive sessions.
One safety gate verifies every action
Every tool call passes through safety.evaluate(). The caller supplies an expect parameter describing the intended outcome. The action executes, then the same call checks whether the UI state matches the expectation. If the UI does not obey, the call returns a DEVIATION report instead of silently continuing. This design catches failed clicks, missed keystrokes, and state mismatches at the point of execution.
Deterministic sequences collapse into a single guarded batch. N tool calls become one round trip, each step still safety-checked. The compact surface exposes seven compound tools covering computer, accessibility, window, system, browser, and task domains. The granular surface exposes individual actions like key_press for finer control. Both surfaces hit the same safety chokepoint.
Performance measured on real tasks
The Windows 11 Calculator test selected a 127 × 49 cell range. clawdcursor completed it in 3 tool calls using roughly 176 tokens of screen input with zero screenshots. The screenshot baseline needed 2 to 8 captures for the same task. Token estimates for clawdcursor derive from character count divided by 3.5; screenshot costs come from Anthropic computer-use documentation. The gap widens on longer workflows because each screenshot loop repeats the full image-to-model round trip.
Six compound tools cover the desktop
The tool surface organizes into six categories. computer provides raw I/O: screenshot, click, double_click, right_click, triple_click, hover, scroll, scroll_horizontal, drag, drag_path, type, key, wait. accessibility drives UI by element name with read_tree, find, get_element, focused, invoke, focus, set_value, get_value, expand, collapse, toggle, select, state, list_children, wait_for. window manages application lifecycle: list, active, focus, maximize, minimize, restore, close, resize, list_displays, screen_size, open_app, open_file, open_url, switch_tab. system handles clipboard, OCR, shortcuts, undo, webview detection, CDP relaunch, and task delegation. browser uses Chrome DevTools Protocol for real DOM access in Electron and WebView2 apps where the a11y tree is sparse. task hands entire workflows to the autonomous loop in daemon mode.
Platform-aware key combos and pause handling
Key combinations map automatically: Cmd on macOS, Ctrl on Windows and Linux. No LLM cost applies to these deterministic calls. When the agent encounters a sign-in screen, 2FA prompt, or CAPTCHA, it pauses instead of guessing. A blue screen-edge glow appears. The user clicks Resume to continue or selects "Something else" to type a response. An amber glow signals active agent operation on all three platforms. Double-clicking the glow stops the agent immediately.
Two run modes for different architectures
The stdio MCP server runs inside your editor. Configuration requires a single JSON block pointing at the clawdcursor binary with mcp --compact arguments. No daemon, no port. The HTTP daemon runs at 127.0.0.1:3847/mcp. agent mode includes a built-in autonomous loop with a configured LLM. agent --no-llm exposes only the tool surface for agents that bring their own reasoning engine. Thirteen-plus model providers are supported.
Installation starts with clawdcursor consent --accept for one-time desktop-control authorization. macOS requires an additional clawdcursor grant for Accessibility and Screen Recording permissions. doctor configures the daemon LLM and runs diagnostics. status verifies consent, permissions, and AI configuration. versions lists every installed copy.
What this means for development teams
Teams automating desktop workflows get predictable token costs, fewer silent failures, and scripts that survive OS updates. The a11y-first approach means automation targets semantic structure rather than visual layout. When the tree is sufficient, the model never sees pixels. When vision is necessary, it activates only for the specific region that needs it. The safety gate turns coordination errors into explicit DEVIATION reports that the calling agent can handle programmatically.
Because the tool surface is model-agnostic, teams can swap providers without rewriting automation logic. The localhost-only default with no telemetry satisfies air-gapped and compliance-sensitive environments. Open-source licensing means the stack can be audited, extended, or self-hosted without vendor lock-in.
The project is available on GitHub. Star the repository to track releases.