skills.rest
perasyudha
perasyudhaCOMMUNITY
Explore all skills by this author →
Website𝕏@Nyxora_AIGitHub

computer-use

Automates desktop GUI interactions via accessibility-tree element indexing across macOS, Windows, and Linux.

5|2|Updated May 26, 2026
Search Tags:#cross-platform#computer use#gui#desktop automation#accessibility tree#cua-driver#element indexing
One-click install
npx skills add https://github.com/perasyudha/Nyxora --skill computer-use-perasyudha
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: computer-use
Source: https://github.com/perasyudha/Nyxora/tree/main/packages/core/playbooks/computer-use
Command: npx skills add https://github.com/perasyudha/Nyxora --skill computer-use-perasyudha

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires cua-driver.

What problem does it solve? Automating a user's real desktop applications normally requires moving the physical cursor, stealing keyboard focus, or switching virtual desktops, which disrupts the user's own work. This Skill drives native apps in the background using accessibility-tree element indices instead of fragile pixel coordinates. ## Core Features & Use Cases - Background GUI Automation: Click, type, scroll, and drag in native apps without moving the user's cursor, grabbing focus, or switching Spaces. - Element-Index Targeting: Capture screenshots with numbered overlays and an AX-tree index, then click elements by index for reliable cross-model operation. - Cross-Platform Support: Works on macOS (AX), Windows (UIA), and Linux (AT-SPI) with any tool-capable model via the cua-driver plumbing. - Use Case: Ask the agent to fill out a form in a native desktop app like Mail or Figma while you keep typing in your editor; the agent captures the app, clicks elements by index, and verifies each step with follow-up screenshots. ## Quick Start Use the computer_use tool to capture the Chrome window and click the Sign In button by its element index.

Frequently Asked Questions about computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate desktop apps without moving the mouse cursor?▼

Use the computer_use tool, which routes input through the accessibility framework instead of the physical cursor. Capture the target app with mode="som" to get numbered element overlays, then click by element index while the user keeps working in other windows.

How to click GUI elements reliably with an AI agent?▼

Capture the screen first with computer_use action="capture" to receive an AX-tree index of interactable elements, then call action="click" with the element number. Element indices are far more reliable than pixel coordinates across different models.

Does computer-use work on Windows and Linux?▼

Yes, it supports macOS, Windows, and Linux through cua-driver, using each platform's native accessibility framework: AX on macOS, UIA on Windows, and AT-SPI on Linux. Keyboard shortcuts differ per platform, such as cmd+s on macOS versus ctrl+s elsewhere.

Why does my element index click fail with a stale element error?▼

SOM element indices are only valid until the next capture, so the UI changing invalidates them. Re-capture the screen before clicking again; the wrapper detects stale element tokens and returns an explicit error instead of clicking the wrong target.

When should I use browser tools instead of computer-use?▼

Use browser_* tools for web automation because they drive a headless Chromium more reliably than a GUI browser. Reserve computer_use for tasks requiring the user's actual native apps, such as Finder, Mail, Figma, or native chat clients.

Related Skills

Explore More Software Engineering Skills→
THTProtocolTHTProtocol

macos-computer-use

Automate macOS mouse and keyboard actions via cua-driver.

Community
Advanced
punkfuncgamespunkfuncgames

record-golden-test

Record deterministic play sessions and capture timestamped keyframe screenshots for golden replay visual regression testing.

Official
Intermediate
NagyViktNagyVikt
2

desktop-commander-ops

Execute local file operations, text editing, and process management via Desktop Commander MCP.

Community
Intermediate
pixiesgeosunpixiesgeosun

playwright-core

Automates web interactions using the Playwright library for Chrome, Firefox, and WebKit.

Community
Intermediate