cua-driver

Automates native macOS apps and browser UI through the local computer-use MCP server.

Updated Jul 7, 2026
One-click install
npx skills add https://github.com/jamsdoescode/Erik-s-Portfolio --skill cua-driver-jamsdoescode
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: cua-driver
Source: https://github.com/jamsdoescode/Erik-s-Portfolio/tree/main/openclicky/AppResources/OpenClicky/OpenClickyBundledSkills/cua-driver
Command: npx skills add https://github.com/jamsdoescode/Erik-s-Portfolio --skill cua-driver-jamsdoescode

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Driving real macOS applications and browser interfaces programmatically is fragile: synthetic clicks steal focus, accessibility trees are sparse, and actions silently fail without verification. This Skill gives agents a disciplined, background-safe loop for operating GUI apps without disrupting the user's foreground work. ## Core Features & Use Cases - Snapshot-Verified Automation: Enforces a snapshot-before-and-after invariant using get_window_state so every click, keystroke, or value change is confirmed against the accessibility tree. - Background-Safe Operation: Launches apps and opens URLs via launch_app without focus steal, cursor warping, or Space switching, so users keep typing in their current app. - Web App Handling: Covers Chromium, WebKit, Electron, and Tauri quirks including sparse AX trees, minimized-window keyboard-commit failures, and the windows-over-tabs pattern. - Use Case: Ask the agent to open a URL in your default browser, fill a form in Numbers, or read your calendar events, and it drives the real app in the background while verifying each step. ## Quick Start Ask the agent to open the Downloads folder in Finder or navigate to a website in your default browser, and it will launch the app, snapshot its window state, act by element index, and verify the result.

Frequently Asked Questions about cua-driver

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate a macOS app without stealing focus?

Use launch_app with the app's bundle id to start it in the background, then call get_window_state to snapshot its accessibility tree and act via element_index clicks. Never use the open CLI or AppleScript activate, which always foreground the target.

How do I open a URL in the user's default browser programmatically?

Resolve the default HTTPS browser bundle id from LaunchServices, then call launch_app with that bundle id and the urls array. This opens a new background window using the user's normal logged-in profile, avoiding the focus-stealing Cmd+L omnibox flow.

Why does pressing Return not work on a minimized browser window?

Keyboard commit events reach the app but renderer focus is not established on minimized windows, so Return fires as a silent no-op or system beep. Use set_value to write the field directly, or AX-click a Submit or Go button instead.

Why do I get 'No cached AX state' errors when clicking elements?

Element indices are cached per (pid, window_id) pair and replaced on every snapshot, so indices from a previous turn or a different window are stale. Call get_window_state with the same window_id in the same turn before acting.

Can this skill record or replay automation trajectories?

No. OpenClicky's managed runtime does not expose recording or replay tools in this build. For regression evidence, save screenshots, logs, and a plain-language step list instead of attempting trajectory capture.

What are the limitations for canvas apps like Figma or Blender?

Canvas and custom-drawn surfaces are not reachable through the accessibility tree, so element_index actions fail. Coordinate clicks are only valid through explicit OpenClicky pixel routes; otherwise the agent should stop and report the missing capability.