skills.rest
xu1713
xu1713COMMUNITY
Explore all skills by this author →
GitHub

macos-computer-use

Automates macOS desktop interactions via screenshots, element indexing, and background input control.

Updated Jun 5, 2026
Search Tags:#accessibility#macos#computer use#screenshot#desktop#gui automation#background input
One-click install
npx skills add https://github.com/xu1713/openhorse --skill macos-computer-use-xu1713
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: macos-computer-use
Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/skills/apple/macos-computer-use
Command: npx skills add https://github.com/xu1713/openhorse --skill macos-computer-use-xu1713

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Automating the macOS desktop usually means hijacking the user's cursor, keyboard focus, and active Space, which interrupts their work. This Skill drives the Mac in the background using the computer_use tool, so the user can keep typing in one app while the AI clicks around in another. ## Core Features & Use Cases - Background GUI Automation: Capture screenshots, click, type, scroll, and drag in any macOS app without stealing cursor focus or switching Spaces. - Element-Indexed Interaction: SOM capture mode overlays numbered indices on every interactable element with an accessibility tree, enabling reliable clicks by index instead of pixel coordinates. - Model-Agnostic Operation: Works with any tool-capable model (Claude, GPT, Gemini, or local OpenAI-compatible endpoints) with no vendor-specific schema. - Use Case: Ask the AI to reply to a message in the native Messages app or organize files in Finder while you continue working in your editor uninterrupted. ## Quick Start Ask the AI to open Safari in the background and click the Sign In button using the computer_use tool.

Frequently Asked Questions about macos-computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS desktop apps without moving the user's cursor?▼

Use the computer_use tool, which routes input to apps in the background without moving the cursor, stealing keyboard focus, or switching Spaces. Capture with app scoping, then click elements by index from the SOM overlay.

How do I click UI elements reliably with an AI agent on macOS?▼

Capture first with mode="som" to get numbered overlays on every interactable element plus an accessibility tree index. Then call click with the element index rather than pixel coordinates, which is far more reliable across models.

Does macOS computer use automation work with models other than Claude?▼

Yes, it works with any tool-capable model including GPT, Gemini, and local models on OpenAI-compatible endpoints. There is no Anthropic-native schema; element-index clicking is recommended since some models are unreliable with raw coordinates.

When should I use browser tools instead of computer_use on macOS?▼

Use browser_* tools for web automation since they run a real headless Chromium and are more reliable than driving the user's GUI browser. Reserve computer_use for native Mac apps like Mail, Messages, Finder, Figma, or games.

Why does computer_use report cua-driver not installed?▼

The cua-driver backend is missing. Run hermes tools and enable Computer Use, which installs cua-driver via its upstream script. macOS Accessibility and Screen Recording permissions are also required.

Why did my element click have no effect on macOS?▼

Element indices come from the last capture and go stale when the UI shifts, such as a new tab or modal dialog. Re-capture to refresh indices, dismiss any blocking modal with escape or its close button, then retry the click.

Related Skills

Explore More Software Engineering Skills→
zaiyemeiyou404zaiyemeiyou404

macos-computer-use

Automate macOS GUI actions via cua-driver without disrupting the desktop.

Community
Advanced
bl1nk-botbl1nk-bot
3

screenshot

Capture desktop screenshots across macOS, Linux, and Windows.

Official
Advanced
patty-chowpatty-chow

macos-computer-use

Drive the macOS desktop with background screenshots, clicks, typing, and scrolling via the computer_use tool.

Community
Intermediate
autonomy-cloudautonomy-cloud
2

macos-automation

Automate native macOS app interactions using AppleScript via osascript.

Official
Intermediate