skills.rest
IceHeartGitH
IceHeartGitHCOMMUNITY
Explore all skills by this author →
GitHub

computer-use

Automates desktop GUI interactions via background clicks, typing, and scrolling across macOS, Windows, and Linux.

1|Updated Jul 22, 2026
Search Tags:#cross-platform#GUI#computer use#desktop automation#accessibility tree#cua-driver#background input
One-click install
npx skills add https://github.com/IceHeartGitH/Hermes-Agent-Android-Termux --skill computer-use-iceheartgith
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: computer-use
Source: https://github.com/IceHeartGitH/Hermes-Agent-Android-Termux/tree/main/custom-skills/autonomous-ai-agents/computer-use
Command: npx skills add https://github.com/IceHeartGitH/Hermes-Agent-Android-Termux --skill computer-use-iceheartgith

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Automating native desktop applications normally requires stealing the user's cursor and keyboard focus, interrupting their work. This Skill drives the desktop in the background using accessibility-tree element targeting, so GUI automation runs without moving the real cursor, switching virtual desktops, or popping windows to the front. ## Core Features & Use Cases - Element-index targeting: Capture screenshots with numbered overlays on every interactable element (SOM mode) plus an accessibility-tree index, then click by element number instead of unreliable pixel coordinates. - Verify-then-escalate ladder: Every input action returns a structured verdict (confirmed, unverifiable, suspected_noop) with escalation recommendations, so the agent climbs from background element clicks to pixel clicks to foreground delivery only when the driver signals it. - Cross-platform input vocabulary: A single action set (capture, click, drag, scroll, type, key, wait, focus_app) works on macOS, Windows, and Linux with any tool-capable model, powered by cua-driver underneath. - Use Case: Ask the agent to fill out a form in a native desktop app like Figma or Mail while you keep typing in your editor; the agent captures the app, clicks fields by index, types values, and verifies each step without ever taking your focus. ## Quick Start Ask the agent to use computer_use to capture the screen of a specific app and click a named button or link in it.

Frequently Asked Questions about computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate desktop app clicks without moving the user's cursor?▼

Use the computer_use capture action with mode="som" to get a screenshot with numbered element overlays, then call click with the element index. cua-driver delivers input in the background, so the real OS cursor never moves and focus is never stolen.

What is the difference between computer_use and browser automation tools?▼

Browser tools drive a headless Chromium and are more reliable for web tasks. computer_use targets the user's actual native apps such as Finder, Mail, Figma, or games, and should be used only when the task requires real desktop applications rather than web pages.

Does computer-use work on Windows, macOS, and Linux?▼

Yes, the same action vocabulary works across all three platforms through cua-driver, which maps to each platform's accessibility framework: AX on macOS, UIA on Windows, and AT-SPI on Linux. Keyboard shortcuts must use the host's idiomatic modifiers like cmd on macOS versus ctrl elsewhere.

Why did my click have no effect on a desktop element?▼

Read the structured verdict returned by the action. If effect is suspected_noop or escalation recommends foreground, retry with coordinate-based clicking, then delivery_mode="foreground". Modal dialogs in Electron apps often block background input and require foreground delivery to dismiss.

When should I not use computer_use for automation?▼

Avoid it for web automation that browser tools can handle, file edits that read_file or write_file cover, and shell commands that belong in a terminal tool. It is meant specifically for driving native GUI applications that have no programmatic interface.

What safety restrictions apply to desktop automation actions?▼

The agent must never click permission dialogs, password prompts, payment UI, or 2FA challenges, and must never type secrets like passwords or API keys. Dangerous shell patterns in typed text and system shortcuts like log out or lock screen are hard-blocked at the tool level.

Related Skills

Explore More Software Engineering Skills→
islee23520islee23520
5

cua-driver

Drive native macOS apps via accessibility tree, mouse, keyboard, and screenshots.

Community
Advanced
ruvnetruvnet
70.1k

browser-replay

Replays recorded browser session trajectories with embedding-based selector recovery for DOM drift.

Community
Advanced
patty-chowpatty-chow

computer-use

Drives desktop applications via background clicking, typing, and scrolling without stealing cursor focus.

Community
Advanced
o-pedrosiqueirao-pedrosiqueira

adaptar-plataforma

Convert Windows PowerShell scripts and Task Scheduler rules to macOS or Linux equivalents.

Community
Advanced