What problem does it solve? Automating native desktop applications normally requires stealing the user's cursor and keyboard focus, interrupting their work. This Skill drives the desktop in the background — clicking, typing, scrolling, and dragging — without moving the real cursor, grabbing focus, or switching virtual desktops, and it works with any tool-capable model. ## Core Features & Use Cases - Element-indexed interaction: Capture screenshots with numbered overlays on every interactable element plus an accessibility-tree index, then click by element index instead of unreliable pixel coordinates. - Verify-and-escalate ladder: Every input action returns a structured verdict (confirmed, unverifiable, suspected_noop) with escalation recommendations from background element clicks to pixel coordinates, typed browser-page routes, and foreground delivery. - Cross-platform support: Works on macOS (AX), Windows (UIA), and Linux (AT-SPI) with platform-appropriate keyboard shortcuts and failure diagnostics via hermes computer-use doctor. - Use Case: Ask the agent to fill out a form in a native desktop app like Figma or Mail while you keep typing in your editor — the agent captures the app window, clicks fields by index, types values, and verifies each step without ever taking over your screen. ## Quick Start Ask the agent to use computer_use to capture the Chrome window and click the Sign In button by its element index.
