macos-computer-use

Use the computer_use tool, which routes input to apps in the background without raising windows, stealing focus, or switching Spaces. Capture the target app first, then click elements by their numbered index from the accessibility tree.

Capture the screen in SOM mode to get numbered overlays on every interactable element, then call click with the element index rather than pixel coordinates. Index-based clicking is far more reliable across models, and you can add capture_after to verify the result in one call.

Yes, the workflow is model-agnostic and works with GPT, Gemini, or open models through local OpenAI-compatible endpoints. There is no Anthropic-native schema; any tool-capable model can use the capture, click, type, and scroll actions.

Use computer_use only when the task requires the user's actual native Mac apps like Finder, Mail, Messages, Figma, or games. For web automation, headless browser tools are more reliable, and for file edits or shell commands, use file and terminal tools directly.

Element indices come from the last capture, so if the UI changed or a modal appeared, the index is stale. Re-capture the app, dismiss any blocking dialog with escape or its close button, then retry the click with the fresh index.

The cua-driver requires macOS with Accessibility and Screen Recording permissions granted. If you see a 'cua-driver not installed' error, run hermes tools and enable Computer Use to install it via the upstream setup script.