skills.rest
baibairui
baibairuiCOMMUNITY
Explore all skills by this author →
GitHub

gateway-desktop

Automate macOS desktop GUI tasks with screenshot-guided actions.

12|3|Updated Mar 6, 2026
Search Tags:#automation#accessibility#macos#screenshot#gui-automation#gateway-desktop
One-click install
npx skills add https://github.com/baibairui/WoClaw --skill gateway-desktop
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gateway-desktop
Source: https://github.com/baibairui/WoClaw/tree/main/.codex/skills/gateway-desktop
Command: npx skills add https://github.com/baibairui/WoClaw --skill gateway-desktop

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires @nut-tree-fork/nut-js, and includes scripts (resource) components.

What problem does it solve?

When tasks require operating desktop applications on macOS and need screenshot-guided GUI actions.

Core Features & Use Cases

  • Start, activate, and control macOS apps using scripted GUI actions with screenshot validation.
  • Support an observe -> act -> observe loop to validate outcomes and drive interactive workflows.
  • Handles local macOS execution, required accessibility and screen recording permissions, and stores visual artifacts under .codex/artifacts/desktop.

Quick Start

Start by running node ./scripts/gateway-desktop.mjs observe to capture the current desktop state.

Frequently Asked Questions about gateway-desktop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS desktop GUI tasks using screenshot validation?▼

Automating macOS desktop GUI tasks with screenshot validation uses an observe-act-observe loop. The Skill captures the current desktop state, executes scripted clicks and typing, and validates the visual outcome to drive interactive workflows.

What macOS permissions are required to run scripted GUI automation?▼

Scripted GUI automation on macOS requires local execution with both accessibility and screen recording permissions. These permissions allow the automation scripts to capture screen states and send control inputs to native applications.

Can I launch and control native macOS apps through visual scripting?▼

Yes, you can launch and control native macOS apps through visual scripting. The automation workflow supports starting, activating, and navigating native applications by applying scripted GUI actions with continuous screenshot validation.

How do I capture the current desktop state before performing GUI actions?▼

To capture the current desktop state before performing GUI actions, run the observe command to initialize the script. This captures a screenshot of the current environment and stores the visual artifacts under the designated desktop directory.

Does this GUI automation approach work with nut-js for native app interactions?▼

Yes, this screenshot-guided GUI automation approach works with nut-js to handle native app interactions. It leverages the library to execute observe-act-observe loops, enabling precise clicking, typing, and screen navigation in real macOS environments.

Related Skills

Explore More Software Engineering Skills→
zaiyemeiyou404zaiyemeiyou404

macos-computer-use

Automate macOS GUI actions via cua-driver without disrupting the desktop.

Community
Advanced
gerald-icagerald-ica

computer-use-agents

Develop AI agents that control computers via vision-based actions.

Community
Advanced
unoplatformunoplatform
9

uno-toolkit-extendedsplashscreen

Overlay custom loading content on the native splash with ExtendedSplashScreen.

Official
Intermediate
aayushsoamaayushsoam
1

macos-computer-use

Automate macOS desktop GUI tasks with screenshots, mouse, keyboard, and scrolling.

Community
Advanced