skills.rest
autonomy-cloud
autonomy-cloudOFFICIAL
Explore all skills by this author →
GitHub

computer-use

Automate macOS desktop workflows by controlling GUI elements via accessibility-tree targeting.

2|2|Updated May 13, 2026
Search Tags:#macos#ui interaction#desktop automation#accessibility tree#apple script#proxy tools#visual workflows
One-click install
npx skills add https://github.com/autonomy-cloud/kairos-interface --skill computer-use-autonomy-cloud
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: computer-use
Source: https://github.com/autonomy-cloud/kairos-interface/tree/main/assistant/src/config/bundled-skills/computer-use
Command: npx skills add https://github.com/autonomy-cloud/kairos-interface --skill computer-use-autonomy-cloud

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It removes the friction of performing repetitive or visual desktop tasks by letting you direct the assistant to observe the screen and interact with macOS UI elements.

Core Features & Use Cases

  • Screen observation with accessibility targeting: Captures the current UI state (including an accessibility tree) so subsequent actions can target elements reliably.
  • End-to-end UI interaction: Click, type text, press keys, scroll, drag-and-drop, and open applications to complete real workflows.
  • Remote execution via proxy: Forwards all actions to a connected desktop client (no local handling of the user’s UI).
  • Use case: Update settings in a desktop app by reading what’s on screen, clicking the right controls via element IDs, entering values, and confirming completion.

Quick Start

Ask the assistant to observe the screen and then click and type in the specific macOS dialog to complete your requested task.

Frequently Asked Questions about computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS desktop tasks by clicking and typing in UI dialogs?▼

macOS desktop automation is achieved by forwarding observe, click, type, and scroll actions to a connected desktop client via proxy tools, targeting UI elements through the accessibility tree to execute visual workflows without CLI or API alternatives.

Can I use Apple Script to interact with macOS UI elements during desktop automation?▼

Yes, Apple Script can be executed as a proxy tool during macOS desktop automation to interact with UI elements, supplementing standard accessibility-tree targeting for clicking, typing, and dragging when native scripting is more effective.

How does accessibility tree targeting work for macOS UI interaction?▼

Accessibility tree targeting captures the current macOS UI state, allowing subsequent actions like clicking, typing, and dragging to reliably target specific elements by their IDs instead of relying on screen coordinates.

Do I need proxy tools to control my macOS desktop remotely?▼

Yes, proxy tools are required to control your macOS desktop remotely, forwarding deterministic coordination commands like observe, click, type, and scroll to a connected desktop client rather than handling the UI locally.

What is the best way to automate repetitive visual workflows on macOS when there is no API?▼

Automating visual workflows without an API is best done using screen-based desktop automation that observes the screen, targets UI elements via the accessibility tree, and executes clicks and data entry through proxy tools.

Can I target multiple macOS clients for desktop automation simultaneously?▼

Yes, multiple macOS clients can be targeted simultaneously during desktop automation by specifying a target_client_id, ensuring deterministic coordination of actions across different connected desktop environments.

Related Skills

Explore More Software Engineering Skills→
vellum-aivellum-ai
1.0k

computer-use

Control a macOS desktop via proxy tools to automate UI tasks.

Official
Intermediate
SSiertsemaSSiertsema

state-transition-mapping

Define and verify complete UI state machines with typed transitions and guards.

Community
Advanced
steipetesteipete
385.7k

peekaboo

Capture and control macOS UI elements via CLI commands.

Community
Advanced
faramirezsfaramirezs

findmy

Tracks Apple devices and AirTags via Find My app using AppleScript and screen capture.

Community
Intermediate