skills.rest
CerisonAutomation
CerisonAutomationCOMMUNITY
Explore all skills by this author →
GitHub

computer-use

Control macOS desktop UI via accessibility tree and element-targeted actions.

Updated Aug 23, 2026
Search Tags:#macos#agentic workflows#desktop automation#gui control#accessibility tree#proxy tools#screen observation
One-click install
npx skills add https://github.com/CerisonAutomation/clud --skill computer-use-cerisonautomation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: computer-use
Source: https://github.com/CerisonAutomation/clud/tree/main/assistant/src/config/bundled-skills/computer-use
Command: npx skills add https://github.com/CerisonAutomation/clud --skill computer-use-cerisonautomation

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates slow, error-prone manual UI interactions by letting the assistant observe the screen and perform precise desktop actions on macOS through accessibility-driven controls.

Core Features & Use Cases

  • Visual screen observation: Capture the current UI state (accessibility tree with stable element IDs) and optionally a screenshot before acting.
  • Direct GUI control: Click, double-click, right-click, type text, press keys/shortcuts, scroll, drag-and-drop, and open or switch apps.
  • Safe session flow: Use explicit completion and response tools to end the computer-use session cleanly after the user goal is met.

Quick Start

Ask the assistant to control your macOS desktop to complete your task by observing the screen, identifying the right UI elements by ID, and then performing the required click/type/keyboard steps.

Frequently Asked Questions about computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS desktop workflows for apps without an API?▼

Yes, you can automate GUI form-filling tasks on macOS by using this Skill to observe the current screen state, identify input fields through the accessibility tree, and execute targeted typing and clicking actions. It handles visual workflows by interacting directly with on-screen elements.

How does AI observe the screen to perform desktop automation?▼

To start automating UI interactions with this Skill, ask the assistant to observe your macOS screen, identify the correct UI elements by their accessibility IDs, and perform the required click, type, or keyboard shortcut steps. It provides proxy-executed tools for direct GUI control and requires a host execution target.

Do I need a host execution target to control macOS desktop actions?▼

Yes, this Skill supports running AppleScript to handle complex macOS desktop automation tasks alongside observing the accessibility tree and executing UI actions. You can use the run AppleScript tool to perform system-level operations that complement standard GUI control workflows.

How do I end a macOS desktop automation session cleanly?▼

To end a macOS desktop automation session cleanly, use the explicit completion and response tools provided by this Skill to trigger a deterministic done/response flow. This ensures the computer-use session terminates safely after your user goal is met.

Related Skills

Explore More Software Engineering Skills→
RuifengFuRuifengFu

cua-driver

Automates macOS app interactions via the Accessibility tree using cua-driver.

Community
Advanced
AcendasAcendas
3

ad-detach

Detach the Android debugger by invoking vm.dispose and removing adb port forwards.

Official
Intermediate
autonomy-cloudautonomy-cloud
2

macos-automation

Automate native macOS app interactions using AppleScript via osascript.

Official
Intermediate
TK0918TK0918

lark-openapi-explorer

Parse official markdown docs to generate lark-cli api commands for undocumented endpoints.

Community
Intermediate