Control macOS GUI apps by clicking OCR-detected numbered elements from screenshots.
npx skills add https://github.com/iBz-04/gloamy --skill mac-use-ibz-04
Please help me install this Agent Skill. Skill: mac-use Source: https://github.com/iBz-04/gloamy/tree/main/skills/mac_use Command: npx skills add https://github.com/iBz-04/gloamy --skill mac-use-ibz-04
It solves the problem of interacting with macOS graphical user interfaces when there is no accessible DOM, API, or reliably identifiable UI hooks—by letting you click and type using visual, OCR-detected elements.
Ask the agent to use mac-use to screenshot your target Mac app window, choose the correct numbered element, perform the click or typing, and then screenshot again to verify the result.
High-intent search queries and answers about installing and using this skill.
You can automate macOS GUI applications visually by taking screenshots, running Apple Vision OCR to detect text, and interacting with numbered clickable regions. This method bypasses missing API hooks to click buttons, scroll, and type inside desktop apps.
Using screenshot-based visual automation is the best way to interact with Mac apps lacking UI hooks. By cropping the app window, detecting text via Apple Vision OCR, and mapping canvas coordinates, you can deterministically click numbered elements, scroll, and type using reliable macOS automation primitives.
Yes, macOS GUI automation with Apple Vision OCR requires specific Python dependencies including pyobjc-framework-Vision for text recognition, pyautogui for clicking and typing, and Pillow for screenshot cropping. These libraries enable the visual element detection and deterministic clicking workflow.
To verify state changes after clicking elements in a Mac app, you perform a verification loop by re-screenshotting the window after each action. This allows you to confirm the visual state change and correct the next interaction before proceeding with further GUI automation steps.
Yes, you can use screenshot-based OCR automation to navigate multi-step workflows in desktop apps. By repeatedly capturing the window, selecting visible text buttons via numbered elements, and typing, you can complete complex tasks like submitting forms or searching inside a browser.
Limitations of using screenshot cropping and OCR for window control include reliance on visible text for detection, meaning unlabelled graphical elements cannot be clicked. Additionally, it requires macOS-only environments and Python dependencies like pyautogui and pyobjc Vision, limiting cross-platform portability.
L-LesterYuAutomate mouse, keyboard, and window management with pyautogui and pygetwindow.
AirmomoAutomate macOS GUI interactions via screenshot-driven OCR and click/type actions.
polyuiislabAutomate macOS GUI interactions using OCR-based element detection and canvas coordinates.
jaminitachiControl macOS UI via Peekaboo CLI and AppleScript for automation.
