computer-use

Automate Windows app UIs using SendInput, UI Automation, and screen capture.

Updated Jul 31, 2026
One-click install
npx skills add https://github.com/AarnavBaddam/skills --skill computer-use-aarnavbaddam
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: computer-use
Source: https://github.com/AarnavBaddam/skills/tree/main/computer-use
Command: npx skills add https://github.com/AarnavBaddam/skills --skill computer-use-aarnavbaddam

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires @oai/sky, and includes scripts (resource) components.

What problem does it solve? Manually operating Windows application interfaces is repetitive and error-prone, and many automation approaches fail when windows are occluded or require fragile coordinate-based clicking. This Skill lets an AI assistant directly control Windows apps through reliable OS-level APIs. ## Core Features & Use Cases - UI Automation: Drive Windows app interfaces using SendInput and UI Automation instead of brittle pixel-based clicking. - Occlusion-Resistant Screenshots: Capture window contents with Windows.Graphics.Capture even when windows are hidden behind others. - Guided Runtime: Bundled documentation topics (guidance, api, confirmations) instruct the agent on target-window workflows, screenshot handling, recovery, and when user confirmation is required. - Use Case: Ask the assistant to open a Windows application, navigate its menus, fill in a dialog, and verify the result via screenshot, all without manual mouse and keyboard work. ## Quick Start Initialize the computer-use runtime in a Node.js session, then ask the assistant to control a specific Windows app such as clicking through a dialog in Notepad.

Frequently Asked Questions about computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate Windows app UI from an AI assistant?

Initialize the computer-use runtime by importing the computer-use-client.mjs script into a Node.js session, which sets up the sky API globally. Then direct the assistant to target a window and perform clicks, typing, and screenshots through UI Automation and SendInput.

How to take screenshots of occluded Windows windows?

Use Windows.Graphics.Capture through the sky API, which captures a window's contents even when it is hidden behind other windows. This avoids the limitations of full-screen capture approaches that only see the visible desktop.

Can I import the @oai/sky runtime directly for Windows automation?

No, you must import the bundled computer-use-client.mjs entry point, which loads the cua_node @oai/sky runtime internally. Direct imports, spawning codex-computer-use.exe, or custom protocol clients bypass app approvals and user interruption handling.

When does Windows UI automation require user confirmation?

The skill includes a confirmations documentation topic that must be read before deciding whether an action needs confirmation. Potentially destructive or irreversible UI actions generally require explicit user approval before execution.

What are the limitations of Windows UI automation with this approach?

Automation depends on the target app exposing UI Automation elements and may fail for apps with custom-rendered interfaces. The guidance documentation covers target-window workflows, screenshot handling, and recovery steps for common failure cases.