skills.rest
lahfir
lahfirCOMMUNITY
Explore all skills by this author →
Website𝕏@mdlahfirGitHub

agent-desktop

Automate desktop application interactions via the agent-desktop CLI.

1.0k|64|Updated Feb 19, 2026
Search Tags:#AI agent#GUI automation#desktop automation#GUI interaction#OS accessibility
One-click install
npx skills add https://github.com/lahfir/agent-desktop --skill agent-desktop-lahfir
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-desktop
Source: https://github.com/lahfir/agent-desktop/tree/main/skills/agent-desktop
Command: npx skills add https://github.com/lahfir/agent-desktop --skill agent-desktop-lahfir

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires agent-desktop, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill allows AI agents to automate interactions with desktop applications, streamlining tasks like clicking buttons, filling forms, and reading UI state.

Core Features & Use Cases

  • Desktop Automation: Control desktop applications through native OS accessibility trees.
  • Observation and Interaction: Click buttons, fill forms, navigate menus, read UI state, toggle checkboxes, scroll, drag, type text, take screenshots, manage windows, and use clipboard.
  • Use Case: An AI agent can use this Skill to automate repetitive tasks in applications like Slack, VS Code, Discord, and Notion.

Quick Start

Use the agent-desktop skill to click the 'OK' button in the 'Save As' dialog of the 'TextEdit' application.

Frequently Asked Questions about agent-desktop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate desktop application interactions using OS accessibility trees?▼

Desktop automation using OS accessibility trees allows AI agents to click buttons, fill forms, and read UI state. This skill uses the agent-desktop CLI to control native desktop applications through their accessibility tree structure.

Can I use GUI automation to control applications like VS Code, Slack, or Notion?▼

Yes, GUI automation can control desktop applications like VS Code, Slack, and Notion. The agent-desktop CLI enables AI agents to interact with these apps by navigating menus, typing text, and managing windows natively.

What desktop automation actions are supported for interacting with UI elements?▼

Desktop automation supports clicking, typing text, scrolling, dragging, toggling checkboxes, and taking screenshots. It also handles window management, clipboard operations, and notifications to fully interact with desktop UI elements.

Do I need the agent-desktop CLI installed to automate desktop applications?▼

Yes, the agent-desktop CLI must be installed to automate desktop applications. This skill relies on the CLI to observe UI state, execute keyboard and mouse actions, and manage the session lifecycle for AI agents.

What is the best way for an AI agent to read UI state and manage windows on a desktop?▼

The best way for an AI agent to read UI state and manage windows is using native OS accessibility trees. This approach captures structural UI data directly, allowing agents to observe elements and execute interaction actions accurately.

Are there limitations when automating GUI interactions through native OS accessibility?▼

GUI automation through OS accessibility is limited by the accessibility tree data exposed by the target application. Applications lacking proper accessibility support may prevent the agent-desktop CLI from accurately observing UI state or executing interactions.

Related Skills

Explore More Software Engineering Skills→
zangjeicyzangjeicy

computer-use

Automate desktop GUI interactions like clicking and typing across macOS, Windows, and Linux.

Community
Advanced
officebeatsofficebeats
8

visual-processor

Extract text, UI components, and relationships from images into Markdown summaries.

Community
Intermediate
hafley66hafley66
1

windows-input-overlay-rust

Intercept Windows input events, extract on-screen text, and render overlays in Rust.

Community
Advanced
MuvonMuvon
3

device-computer-automation

Controls mouse, keyboard, and screen capture via desktop-automation MCP server for GUI tasks.

Official
Intermediate