mac-use

Automate macOS GUI interactions using OCR-based element detection and canvas coordinates.

1.2k|53|Updated Dec 24, 2025

Search Tags:#automation #macos #scripting #window-management #gui-automation #vision-ocr

Ready to run

npx skills add https://github.com/polyuiislab/infiAgent --skill mac-use

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: mac-use
Source: https://github.com/polyuiislab/infiAgent/tree/main/skills/mac-use
Command: npx skills add https://github.com/polyuiislab/infiAgent --skill mac-use

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyobjc-framework-Vision, pyautogui, Pillow, and includes scripts (resource) components.

What problem does it solve?

Automate repetitive GUI interactions on macOS by recognizing on-screen text and providing a reliable, element-based way to interact with apps.

Core Features & Use Cases

OCR-based text detection and element identification in macOS apps
Numbered element clicking, typing, scrolling, and key presses across macOS windows
Robust window activation, coordinate mapping, and fallback canvas coordinates for unlabeled icons
Use cases include automating routine UI tasks, data entry, and GUI testing on macOS

Quick Start

Open a macOS app, run the screenshot command to detect elements, then click a numbered element or type text.

Frequently Asked Questions about mac-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate macOS GUI interactions without accessibility identifiers?▼

You can automate macOS GUI interactions using OCR-based element picking. The Skill detects on-screen text via Vision OCR, maps numbered elements on a 1000x1000 canvas, and executes scriptable clicks and typing for apps lacking accessibility labels.

Can I click buttons and type text in desktop apps using Python scripting?▼

Yes, you can click numbered elements, type text, scroll, and press keys in desktop apps. Through a dedicated scripts directory, it applies PyAutoGUI and Vision OCR to interact with macOS windows after taking a screenshot to detect UI elements.

Does this OCR-driven automation work for unlabeled icons in macOS apps?▼

OCR-driven automation handles unlabeled icons using a fallback 1000x1000 canvas coordinate system. When Vision OCR cannot detect text on an icon, you can navigate and interact using mapped canvas coordinates instead of text labels.

What do I need to set up to start automating macOS windows?▼

You need a macOS environment with the required dependencies installed, including pyobjc-framework-Vision, pyautogui, and Pillow. Open a target macOS app, run the screenshot command to detect elements, then execute clicks or typing.

What are the limitations of using Vision OCR for GUI testing on macOS?▼

Vision OCR for GUI testing relies on on-screen text recognition, meaning purely graphical elements without text require fallback canvas coordinates. It is best suited for routine UI tasks, data entry, and testing rather than pixel-perfect image recognition.