skills.rest
manaflow-ai
manaflow-aiOFFICIAL
Explore all skills by this author →
GitHub

cmux-cua

Drive macOS GUI apps from agent sessions via accessibility tree and screenshot perception.

26.6k|2.3k|Updated Jan 28, 2026
Search Tags:#screenshots#mcp#computer use#gui automation#Accessibility Tree#Agent#MacOS
One-click install
npx skills add https://github.com/manaflow-ai/cmux --skill cmux-cua
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: cmux-cua
Source: https://github.com/manaflow-ai/cmux/tree/main/skills/cmux-cua
Command: npx skills add https://github.com/manaflow-ai/cmux --skill cmux-cua

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

AI coding agents running in a terminal cannot see or operate real macOS GUI applications. This Skill gives agents computer-use capabilities—reading the accessibility tree, capturing screenshots, and performing clicks, typing, scrolling, and dragging—through the cmux-cua engine bundled with the cmux terminal.

Core Features & Use Cases

  • GUI Perception and Control: Inspect app UI via the accessibility tree, take screenshots, and act with click, type_text, press_key, scroll, and drag tools, with a branded agent cursor showing each action.
  • Dual Agent Profiles: Attaches automatically as an MCP tool server to Claude Code and Codex sessions launched inside cmux, with a ten-tool Codex-compatible profile and an extended native cmux profile.
  • Managed Permissions and Sessions: Handles one-time Accessibility and Screen Recording grants through a dedicated helper app, plus a menu-bar item to focus the driven app or the calling terminal.
  • Use Case: Ask your agent to open Calculator and click 100 + 105 button by button; the agent snapshots the app state, clicks each on-screen control with the visible branded cursor, and verifies the result from a fresh screenshot.

Quick Start

Start a Claude or Codex session inside cmux and ask the agent to open a macOS app and operate its on-screen controls using the computer-use tools.

Frequently Asked Questions about cmux-cua

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I give an AI agent control of macOS apps?▼

Launch Claude Code or Codex inside the cmux terminal and the cmux-cua computer-use tools attach automatically as an MCP server. The agent can then read the accessibility tree, take screenshots, and click, type, scroll, or drag in real macOS apps.

What permissions does macOS computer use require?▼

Two one-time grants are required, both owned by the cmux Computer Use helper app rather than cmux itself: Accessibility for inspecting and driving UI, and Screen Recording for screenshots. Onboarding appears on the first tool invocation and opens the matching System Settings panes.

Why are my agent's screenshots black or empty?▼

Blank screenshots mean Screen Recording permission has not been granted to the cmux Computer Use helper. Grant it in Settings → Computer Use, let the helper daemon refresh, then retry the tool call.

Why do computer-use clicks fail with a permission error?▼

Click failures with permission errors indicate Accessibility has not been granted to the cmux Computer Use helper. Grant it through Settings → Computer Use, wait for the helper to pick up the grant, and retry the action.

Does cmux-cua work with Codex and Claude Code?▼

Yes, both are supported with different tool profiles. Codex receives a ten-tool roster matching its built-in Computer Use connector, while Claude gets the extended native profile with element tokens, batched perform_actions, and window-state snapshots.

Why does a TCC prompt mention Codex Computer Use?▼

That prompt comes from the codex CLI's own helper, not cmux. macOS attributes the request to the responsible parent process, so it appears as cmux asking to control Codex Computer Use; denying it does not affect cmux computer use.

Related Skills

Explore More Software Engineering Skills→
valentinuuiuiuvalentinuuiuiu

findmy

Locate lost Apple devices and AirTags via the Find My app on macOS.

Community
Intermediate
imbue-aiimbue-ai
401

asciinema-demos

Create concise asciinema demo recordings that visually verify completed work.

Official
Advanced
indent-comindent-com
10

blit

Automate terminal multiplexing and Wayland GUI interaction via the blit CLI.

Official
Intermediate
MarbleSodasMarbleSodas

findmy

Automate Find My via AppleScript screenshots and vision analysis to extract device locations.

Community
Intermediate