skills.rest
Airmomo
AirmomoCOMMUNITY
Explore all skills by this author →
GitHub

mac-use

Automate macOS GUI interactions via screenshot-driven OCR and click/type actions.

Updated Mar 24, 2026
Search Tags:#automation#ocr#macos#vision#screenshot#openclaw#gui-automation
One-click install
npx skills add https://github.com/Airmomo/skills --skill mac-use-airmomo
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: mac-use
Source: https://github.com/Airmomo/skills/tree/main/mac-use
Command: npx skills add https://github.com/Airmomo/skills --skill mac-use-airmomo

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyobjc-framework-Vision, pyautogui, Pillow, and includes scripts (resource) components.

What problem does it solve?

Automates macOS GUI tasks by enabling an AI agent to interact with desktop applications through a screenshot → OCR → click/type loop, eliminating repetitive manual operations.

Core Features & Use Cases

  • Screenshots windows, runs local OCR to detect text, and annotates UI elements for deterministic interaction.
  • Clicks, types, and scrolls inside target applications, with verification steps to ensure the expected result.
  • Use Case: Automate opening a macOS app, locating a button by text, and performing a sequence of actions to complete a workflow with minimal human input.

Quick Start

Open the target macOS app, run a screenshot to detect text, then click an element by its number using clicknum.

Frequently Asked Questions about mac-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS GUI interactions using screenshots?▼

You can automate macOS GUI interactions by using a screenshot-driven loop that captures the screen, runs local OCR to detect text, and executes clicks or typing. This approach automates desktop apps without needing direct API access.

Does macOS GUI automation require external cloud APIs for OCR?▼

No, macOS GUI automation can rely entirely on local Apple Vision OCR. It processes screenshots directly on-device to detect UI text elements, ensuring privacy and eliminating the need for external cloud API dependencies.

Can I use PyAutoGUI to click specific UI elements by text on macOS?▼

Yes, you can use PyAutoGUI for input actions like clicking and typing. The Skill locates specific UI elements by running local Apple Vision OCR on screenshots, then maps the detected text to coordinates for PyAutoGUI to interact with.

What is the best way to locate and click a button in a macOS app by its text?▼

The best way is taking a screenshot of the target app window, running Apple Vision OCR to detect text and annotate UI elements, then using the assigned element number to click the button. This ensures deterministic interaction.

How do I verify if an automated click or typing action succeeded on macOS?▼

You verify automated actions by taking a subsequent screenshot of the macOS desktop. The system checks the updated UI state through local OCR to confirm the expected result of the click or typing action.

Do I need to install Quartz to handle windows for macOS desktop automation?▼

Yes, Quartz is required for window handling in macOS desktop automation. It works alongside local Apple Vision OCR and PyAutoGUI to target specific application windows during the screenshot and interaction process.

Related Skills

Explore More Software Engineering Skills→
iBz-04iBz-04
63

mac-use

Control macOS GUI apps by clicking OCR-detected numbered elements from screenshots.

Community
Intermediate
polyuiislabpolyuiislab
1.2k

mac-use

Automate macOS GUI interactions using OCR-based element detection and canvas coordinates.

Community
Advanced
mikeyobrienmikeyobrien
369

clipboard

Read and write macOS clipboard contents using pbpaste and pbcopy.

Community
Basic
nnancennance

managing-apple-services

Manage Apple Calendar, Contacts, and Notes via shell scripts and Swift CLI.

Community
Intermediate