skills.rest
Muvon
MuvonOFFICIAL
Explore all skills by this author →
Website𝕏@muvonteamGitHub

device-computer-automation

Controls mouse, keyboard, and screen capture via desktop-automation MCP server for GUI tasks.

3|Updated Mar 14, 2026
Search Tags:#multi-monitor#screen capture#accessibility permissions#window focus#desktop ui automation#mouse and keyboard#pixel targeting
One-click install
npx skills add https://github.com/Muvon/octomind-tap --skill device-computer-automation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: device-computer-automation
Source: https://github.com/Muvon/octomind-tap/tree/main/skills/device-computer-automation
Command: npx skills add https://github.com/Muvon/octomind-tap --skill device-computer-automation

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Operates your local desktop’s native user interface by taking targeted screenshots and issuing mouse, keyboard, and clipboard actions when no reliable UI semantic tree or automation API is available.

Core Features & Use Cases

  • Pixel-anchor desktop targeting: Click and type by visual anchors and region-based recognition rather than accessibility IDs.
  • Desktop-automation MCP control: Uses the desktop-automation server to move/click, type keys, press shortcuts, and capture screen regions.
  • Safety and reliability guardrails: Enforces a verification loop (snapshot → locate → act → verify → record) and prevents risky actions like screen-lock/biometric bypass or destructive system changes.

Quick Start

Ask the agent to automate a native action on your desktop for a specific window, such as taking you through saving a file from the currently focused app while confirming the correct window focus first.

Frequently Asked Questions about device-computer-automation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate desktop UI tasks when no accessibility IDs or semantic tree are available?▼

You can automate native desktop GUIs by using pixel-anchor targeting, which takes screenshots to locate visual anchors and then issues mouse, keyboard, and clipboard actions without relying on a UI semantic tree or accessibility IDs.

Does desktop automation work with multi-monitor setups on macOS, Linux, and Windows?▼

Yes, desktop automation supports macOS, Linux, and Windows workflows with multi-monitor awareness, allowing you to target specific screen regions and capture areas across multiple displays.

Do I need special OS permissions to control mouse and keyboard actions on my desktop?▼

Yes, you must grant OS input and accessibility permissions to allow the desktop-automation server to control the mouse, keyboard, and screen capture functions on your operating system.

What is the verification loop for safe desktop UI automation?▼

The safety verification loop follows a snapshot, locate, act, verify, and record sequence to ensure screenshot-verified actuation, preventing risky actions like screen-lock bypass or destructive system changes.

Are there limitations on region capture size for desktop automation workflows?▼

Yes, region-capture discipline enforces response-size limits to manage payload constraints, meaning you must target specific screen regions rather than capturing full unrestricted screenshots.

Can I use pixel targeting to automate saving a file from the currently focused window?▼

Yes, you can ask the agent to automate native actions like saving a file, and the automation will first confirm the correct window focus before executing mouse and keyboard actions via pixel anchors.

Related Skills

Explore More Software Engineering Skills→
hungthinh04hungthinh04

macos-computer-use

Control macOS desktop apps in the background without stealing focus.

Community
Advanced
AIDotNetAIDotNet
84

screenshot-capture

Capture full screens, windows, or regions and save as PNG files.

Official
Basic
EdanStarfireEdanStarfire
15

process-manager

Manage processes by PID across Windows, Linux, and macOS.

Community
Basic
autonomy-cloudautonomy-cloud
2

computer-use

Automate macOS desktop workflows by controlling GUI elements via accessibility-tree targeting.

Official
Advanced