What problem does it solve? Driving a user's desktop applications normally requires moving the real cursor and stealing keyboard focus, which interrupts the user's own work. This Skill lets an agent click, type, scroll, and drag in the background using accessibility-tree element indices instead of fragile pixel coordinates, on any major desktop platform. ## Core Features & Use Cases - Background GUI automation: Click, double-click, drag, scroll, type, and send keyboard shortcuts without raising windows, moving the OS cursor, or switching virtual desktops. - Element-index targeting: Capture screenshots with numbered overlays plus an accessibility-tree index, then act on elements by index for far higher reliability than raw coordinates. - Cross-platform support: Works on macOS (AX), Windows (UIA), and Linux (AT-SPI) with platform-appropriate keyboard shortcuts and failure-mode guidance. - Use Case: Ask the agent to fill out a form in a native desktop app like Mail or Figma while you keep typing in your editor; the agent captures the app, clicks fields by element index, and verifies each step with follow-up captures. ## Quick Start Ask the agent to capture the screen of a specific app and click a named button in it using the computer_use tool.
