skills.rest
AtlasOmnia
AtlasOmniaCOMMUNITY
Explore all skills by this author →
Website𝕏@AtlasOmniaGitHub

computer-use

Drives desktop applications via background clicking, typing, and scrolling across macOS, Windows, and Linux.

115|9|Updated Aug 5, 2026
Search Tags:#cross-platform#computer use#gui#desktop automation#accessibility tree#cua-driver#background input
One-click install
npx skills add https://github.com/AtlasOmnia/donna-starter --skill computer-use-atlasomnia
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: computer-use
Source: https://github.com/AtlasOmnia/donna-starter/tree/main/skills/computer-use
Command: npx skills add https://github.com/AtlasOmnia/donna-starter --skill computer-use-atlasomnia

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Automating native desktop applications normally requires stealing the user's cursor and keyboard focus, interrupting their work. This Skill drives the desktop in the background using accessibility-tree element indices, so GUI automation runs without moving the real cursor, grabbing focus, or switching virtual desktops. ## Core Features & Use Cases - Background GUI automation: Capture screens with numbered element overlays (SOM mode), then click, type, scroll, and drag by element index without raising windows or switching Spaces. - Cross-platform support: Works on macOS (Accessibility API), Windows (UIA), and Linux (AT-SPI) with any tool-capable model, powered by cua-driver under the hood. - Safety guardrails: Hard rules against clicking permission dialogs, typing secrets, or following instructions found in screenshots, plus blocked dangerous shell patterns. - Use Case: Ask the agent to organize files in Finder, fill a form in a native desktop app, or operate Figma while you keep typing in your editor uninterrupted. ## Quick Start Ask the agent to open a specific desktop application and perform a task in it, such as organizing your Downloads folder in Finder without interrupting your current work.

Frequently Asked Questions about computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate desktop apps without moving the mouse cursor?▼

Use the computer_use tool, which routes input through the platform accessibility API instead of the OS cursor. Capture the target app with mode="som" to get numbered element overlays, then click by element index while the real cursor stays untouched.

How do I click UI elements reliably with computer_use?▼

Capture the screen first with action="capture" and mode="som" to get an indexed accessibility tree, then call action="click" with the element number. Element indices are far more reliable than pixel coordinates, but they expire after each new capture.

Does computer-use work on Windows and Linux or only macOS?▼

It works on macOS, Windows, and Linux through cua-driver, using each platform's accessibility framework: AX on macOS, UIA on Windows, and AT-SPI on Linux. Keyboard shortcuts differ per platform, such as cmd+s on macOS versus ctrl+s elsewhere.

When should I use browser tools instead of computer-use?▼

Use browser_* tools for web automation because they drive a headless Chromium more reliably than a GUI browser. Reserve computer_use for native applications like Finder, Mail, Figma, or chat clients that browser tools cannot reach.

Why does computer-use capture return empty or no window errors?▼

On Linux, DISPLAY may be unset or you may be on pure Wayland; on Windows, you may be in Session 0 via SSH rather than the interactive desktop. Run hermes computer-use doctor to get a structured diagnostic report of the exact cause.

What actions are blocked by computer-use safety rules?▼

The tool refuses to click permission dialogs, password prompts, payment UI, or 2FA challenges, and never types passwords, API keys, or credit card numbers. Dangerous shell patterns in typed text and shortcuts like log out or lock screen are hard-blocked.

Related Skills

Explore More Software Engineering Skills→
MarbleSodasMarbleSodas

hermes-agent

Configure and extend Hermes Agent for autonomous terminal workflows across tools and sessions.

Community
Intermediate
ruvnetruvnet
70.1k

browser-replay

Replays recorded browser session trajectories with embedding-based selector recovery for DOM drift.

Community
Advanced
cloud-native-toolscloud-native-tools

browser-utils

Automate web browser interactions using native tools, MCP connectors, or Playwright scripts.

Official
Advanced
marvelousempiremarvelousempire

multi-surface-single-engine

Centralize operation logic in one script, API, or binary with multiple interfaces.

Community
Advanced