skills.rest
wanhaoniu
wanhaoniuCOMMUNITY
Explore all skills by this author →
GitHub

macos-use-desktop-control

Automate macOS desktop UI interactions by locating elements via visible text.

41|2|Updated Feb 17, 2026
Search Tags:#automation#ui#accessibility#macos#desktop-control#text-based-search
One-click install
npx skills add https://github.com/wanhaoniu/MomoAgent --skill macos-use-desktop-control
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: macos-use-desktop-control
Source: https://github.com/wanhaoniu/MomoAgent/tree/main/skills/macos-use-desktop-control
Command: npx skills add https://github.com/wanhaoniu/MomoAgent --skill macos-use-desktop-control

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

macOS desktop automation with reliability by locating elements using visible text rather than fixed coordinates, enabling robust interactions across apps.

Core Features & Use Cases

  • Open or activate macOS applications and traverse their accessibility UI trees.
  • Find elements by visible text and click, with optional role filtering and typing.
  • Type text, press keys, and scroll within the targeted UI, without hard-coded coordinates.
  • Suitable for UI testing, workflow automation, and rapid prototyping of macOS desktop interactions.

Quick Start

Open an app and click the menu item labeled 'Open' by text using the macos-use-desktop-control tool.

Frequently Asked Questions about macos-use-desktop-control

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS desktop UI interactions without using fixed coordinates?▼

You can automate macOS desktop UI interactions by locating elements through their visible text and accessibility roles. This approach enables robust clicking, typing, and scrolling across apps without relying on hard-coded screen coordinates.

Do I need special permissions to control macOS applications via accessibility?▼

Yes, you need macOS accessibility permissions granted to the controlling application. You also need the macos-use MCP server installed and running to execute the text-based UI element search and interaction commands.

What is the best way to click a specific menu item in a macOS app using automation?▼

The best way to click a macOS menu item is by searching for the element using its visible text label. This allows the automation tool to traverse the app's accessibility UI tree and perform the click action reliably.

Can I use text-based search to find and interact with UI elements in any macOS app?▼

Text-based search works with any macOS app that has accessibility features enabled. The tool locates UI elements by matching visible text and optional role filters, allowing you to target buttons, menus, and input fields.

Why does my macOS UI automation script fail to find elements after app launch?▼

macOS UI automation may fail if the target app lacks accessibility permissions or the UI tree is not fully loaded. Ensure the macos-use MCP server is running and the app supports accessibility traversal for text-based element location.

Does macos-use-desktop-control support typing text and pressing keys within targeted apps?▼

Yes, macos-use-desktop-control supports typing text, pressing keys, and scrolling within the targeted UI. Once an element is located using visible text, you can perform these robust actions without hard-coded coordinates.

Related Skills

Explore More Software Engineering Skills→
RuifengFuRuifengFu

cua-driver

Automates macOS app interactions via the Accessibility tree using cua-driver.

Community
Advanced
JoeyJoziahJoeyJoziah

peekaboo

Automate macOS UI capture, inspection, and interaction workflows via the Peekaboo CLI.

Community
Advanced
Probably-GroupProbably-Group
3

macos-accessibility

Inspect and control macOS UI elements via AXUIElement with permission checks.

Official
Advanced
Soul-Brews-StudioSoul-Brews-Studio
116

physical

Retrieve Nat's current coordinates from FindMy CSV data with map links and device statuses.

Official
Advanced