What problem does it solve? Automating desktop applications usually requires moving the user's cursor and stealing keyboard focus, interrupting their work. This Skill drives the desktop in the background using accessibility-tree element indices instead of fragile pixel coordinates, so automation runs while the user keeps working. ## Core Features & Use Cases - Background GUI Automation: Capture screens with numbered element overlays (SOM mode), then click, type, scroll, and drag by element index without moving the user's cursor or raising windows. - Verify-and-Escalate Workflow: Every input action returns a structured verdict (confirmed, unverifiable, suspected_noop) with an escalation ladder from background element clicks to pixel coordinates to foreground delivery. - Cross-Platform Support: Works on macOS (AX), Windows (UIA), and Linux (AT-SPI) with any tool-capable model, powered by cua-driver under the hood. - Use Case: Ask the agent to fill out a form in a native desktop app like Figma or Mail while you continue typing in your editor; the agent captures the app, clicks fields by index, and verifies each step. ## Quick Start Use the computer_use tool to capture the Chrome window and click the Sign In button by its element index.
