llm-computer-use

Automate Windows desktop clicks, keystrokes, and window management via screen observation.

Updated Aug 27, 2026

Search Tags:#llm #anthropic #windows #screen-capture #pyautogui #desktop-automation

Ready to run

npx skills add https://github.com/karstenheld3/SharePoint-GPT-Middleware --skill llm-computer-use

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llm-computer-use
Source: https://github.com/karstenheld3/SharePoint-GPT-Middleware/tree/main/.windsurf/skills/llm-computer-use
Command: npx skills add https://github.com/karstenheld3/SharePoint-GPT-Middleware --skill llm-computer-use

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill enables automation of repetitive desktop interactions by letting an AI agent observe the screen, decide on actions, and perform clicks, keystrokes, and window operations.

Core Features & Use Cases

Vision-based desktop automation: AI analyzes on-screen elements to trigger clicks, typing, and window management.
CLI and programmatic usage: Run tasks from the command line or embed into Python scripts for batch automation.
Safe execution modes: Supports dry-run to validate workflows before performing actions on your machine.

Quick Start

Open Notepad and type Hello World.

Frequently Asked Questions about llm-computer-use

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate desktop tasks with AI vision on Windows?▼

Desktop automation with AI vision lets an AI agent view your screen and perform clicks, keystrokes, and window management. This Skill enables that by combining screen capture, input simulation, and Claude's vision capabilities to automate repetitive tasks like opening applications, filling fields, and navigating file systems on Windows.

Can I automate browser and application interactions without writing complex scripts?▼

Yes. This Skill abstracts the complexity of screen observation and action sequencing. You define the task (e.g., open Notepad and type text), and the AI agent handles analyzing the screen and executing the necessary clicks and keystrokes, requiring only Python 3.10+ and an Anthropic API key.

What are the system requirements for desktop automation with PyAutoGUI and screen capture?▼

You need Windows OS, Python 3.10 or higher, an Anthropic API key, and libraries for screen capture and input simulation. The Skill handles screen observation and command execution, so no additional desktop automation frameworks are required beyond what's built in.

How do I validate automation workflows before running them on my machine?▼

This Skill supports dry-run mode, which lets you test and validate workflows without performing actual clicks, keystrokes, or window operations. This allows you to review the AI agent's planned actions before committing them to your desktop.

Can I run desktop automation from the command line or embed it in Python scripts?▼

Both. The Skill supports CLI usage for one-off tasks and programmatic integration into Python scripts for batch automation. This flexibility lets you automate desktop interactions standalone or as part of larger automation workflows.

What are the limitations of vision-based desktop automation on Windows?▼

Vision-based automation depends on screen visibility and UI clarity. Performance scales with screen complexity, and actions that require sub-pixel precision or interactions outside the visible desktop may face constraints. The Skill works best for standard application workflows with clear UI elements.