skills.rest
autonomy-cloud
autonomy-cloudOFFICIAL
Explore all skills by this author →
GitHub

app-control

Drive macOS applications via raw keyboard and mouse input.

2|2|Updated May 13, 2026
Search Tags:#accessibility#macos#ui automation#keyboard shortcuts#emulators#raw input#window-relative clicking
One-click install
npx skills add https://github.com/autonomy-cloud/kairos-interface --skill app-control-autonomy-cloud
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: app-control
Source: https://github.com/autonomy-cloud/kairos-interface/tree/main/assistant/src/config/bundled-skills/app-control
Command: npx skills add https://github.com/autonomy-cloud/kairos-interface --skill app-control-autonomy-cloud

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When a macOS app’s Accessibility (AX) tree is missing, incomplete, or unreliable, humans waste time trying to operate it manually because the assistant cannot reliably “see” UI elements and interact through standard UI navigation.

Core Features & Use Cases

  • Proxy tool access for a single app: Exposes app_control_* tools to drive one named macOS application via raw input (keyboard, mouse, screenshots) rather than AX-based interaction.
  • Reliable observation before acting: Captures lifecycle and window state via app_control_observe, with a configurable settle delay to avoid “one input behind” screenshots.
  • Precision input primitives: Supports app_control_press, app_control_combo, app_control_sequence, app_control_type, plus window-relative app_control_click and app_control_drag.
  • Session control: Uses app_control_start to begin a targeted session and app_control_stop to end the session without auto-quit.

Quick Start

Use the app-control skill to start an app-control session for Safari and send a sequence of key presses to navigate to the address bar.

Frequently Asked Questions about app-control

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS UI when the Accessibility tree is empty or unhelpful?▼

Automate macOS UI without an Accessibility tree by using raw keyboard and mouse input to drive applications, sending key presses and window-relative clicks to interact with elements directly.

Can I use raw input to automate games and emulators on macOS?▼

Yes, you can automate games and emulators on macOS using raw input. This approach sends direct keyboard and mouse commands to interact with custom-rendered canvases where standard UI navigation fails.

How do I click elements inside an OpenGL or Electron app using coordinates?▼

Click elements inside OpenGL or Electron apps by using window-relative coordinates. This method bypasses the unreliable Accessibility tree and targets specific screen positions within the application window.

Why are my macOS automation screenshots delayed by one input?▼

Screenshots in macOS automation appear delayed by one input when window state changes are captured too quickly. Applying a configurable settle delay during observation prevents capturing stale window states.

What is the best way to manage an active session for raw input automation?▼

Manage a raw input automation session by explicitly starting a targeted session for a named application, sending input commands, and ending with a stop action to close the session without auto-quitting the app.

Related Skills

Explore More Software Engineering Skills→
Project-N-E-K-OProject-N-E-K-O
2.4k

3d-camera-interaction

Align mouse input with 3D model drag and zoom in Three.js.

Official
Advanced
X-School-AcademyX-School-Academy
9

dev-swarm-use-computer

Automate GUI interactions and screen capture for UI testing.

Official
Intermediate
WarrenZhu050413WarrenZhu050413
7

Sending Notifications

Send macOS modal notifications from tmux terminal scripts via osascript.

Community
Basic
autonomy-cloudautonomy-cloud
2

computer-use

Automate macOS desktop workflows by controlling GUI elements via accessibility-tree targeting.

Official
Advanced