skills.rest
shiwenwen
shiwenwenCOMMUNITY
Explore all skills by this author →
GitHub

ha-mac-control

Automate macOS desktop control via mac_control actions with UI state recovery.

1.4k|129|Updated Mar 13, 2026
Search Tags:#accessibility#macos#ui verification#desktop automation#visual targeting#dock and spaces#clipboard handling
One-click install
npx skills add https://github.com/shiwenwen/hope-agent --skill ha-mac-control
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ha-mac-control
Source: https://github.com/shiwenwen/hope-agent/tree/main/skills/ha-mac-control
Command: npx skills add https://github.com/shiwenwen/hope-agent --skill ha-mac-control

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes assets (resource) components.

What problem does it solve?

It solves the difficulty of reliably operating macOS UI elements (apps, windows, dialogs, menus, Dock, Spaces, and clipboard) when accessibility identifiers are volatile and the UI state changes between steps.

Core Features & Use Cases

  • Authorized macOS desktop control with state recovery: performs status checks, app/window/menu/dialog inspection, and action execution with fresh observation to avoid stale UI targets.
  • Robust targeting rules for UI automation: uses bundleId/windowId/snapshotId + elementId re-resolution and avoids guessing when multiple elements match.
  • Visual positioning and OCR when AX is insufficient: locates visible UI via annotated visual observe, OCR, and pixel-to-screen point mapping, then verifies results after actions.
  • Safe workflow verification loops: re-snapshots, re-lists, or inspects dialogs/menus after mutations to confirm the expected state.

Quick Start

Ask the AI to control your Mac to open a specific app, navigate to the UI location you describe (e.g., a Dock item, a Space, a menu command, a dialog button, or a visible button), and then perform the action while it verifies the result.

Frequently Asked Questions about ha-mac-control

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS desktop control when accessibility identifiers change between steps?▼

Automating macOS desktop control uses fresh observation and snapshotId/elementId re-resolution to target volatile UI elements. It safely clicks, types, and navigates local apps, Dock, Spaces, and dialogs while verifying state after each action.

What is the best way to click a specific button in a Mac app when accessibility identifiers are insufficient?▼

Visual targeting and OCR locate visible UI elements via annotated visual observe and pixel-to-screen point mapping. This approach identifies on-screen buttons when standard accessibility identifiers fail or return ambiguous matches.

Does macOS UI automation work with switching Spaces and managing the Dock?▼

Yes, macOS UI automation supports authorized control of the Dock and Spaces. You can instruct the AI to switch between Spaces, launch or focus windows, and interact with Dock items while verifying the expected state.

Why does my Mac automation script fail to find the correct dialog button?▼

Mac automation scripts fail when targeting stale UI state or when multiple elements match a query. The workflow avoids guessing by using dialog.inspect, re-snapshotting, and re-listing windows to confirm the expected state after actions.

Can I use macOS desktop automation for safe clipboard handling and app navigation?▼

Yes, macOS desktop automation includes clipboard-safe workflows and app navigation. It performs status checks, app window focusing, and menu command execution while applying verification-oriented recovery loops to ensure safe operations.

Related Skills

Explore More Software Engineering Skills→
whilpwhilp
9

cleanshot

Automate screenshots, recordings, scrolling captures, and OCR via CleanShot X CLI.

Community
Advanced
weisbergweisberg
2

clipli

Processes clipboard text and images with commands for converting and saving templates.

Community
Intermediate
silva2kandsilva2kand

agentic-image-loop

Automate iterative image generation, annotation, and refinement workflows.

Community
Intermediate
storyclaw-officialstoryclaw-official

peekaboo

Automate macOS UI interactions and screen captures with the Peekaboo CLI tool.

Official
Advanced