skills.rest
mlamla123
mlamla123COMMUNITY
Explore all skills by this author →
GitHub

desktop-vision

Automate macOS desktop observation and interaction using native tools and vision models.

Updated Aug 27, 2026
Search Tags:#automation#macos#osascript#screencapture#vision-models#cliclick#desktop-vision
One-click install
npx skills add https://github.com/mlamla123/openclaw-starter-kit --skill desktop-vision
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: desktop-vision
Source: https://github.com/mlamla123/openclaw-starter-kit/tree/main/skills/desktop-vision
Command: npx skills add https://github.com/mlamla123/openclaw-starter-kit --skill desktop-vision

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires opencv-python, Pillow, numpy, requests, websocket-client.

What problem does it solve?

See and interact with the macOS desktop using a vision-enabled workflow to observe content, detect UI elements, and perform scripted actions without relying on browser automation.

Core Features & Use Cases

  • Screen capture and analysis using native macOS tools + vision models to understand on-screen state.
  • Native interactions: clicking, typing, and window management via CLI utilities and AppleScript.
  • Use Case: automate routine UI checks, dismiss dialogs, and perform long-running monitoring of apps.

Quick Start

Open your OpenClaw workspace and start a desktop-vision session to watch and interact with your macOS desktop.

Frequently Asked Questions about desktop-vision

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate macOS desktop interaction without browser automation?▼

Automate macOS desktop interaction by capturing screen content and detecting UI elements with vision models, then performing native clicks and typing via osascript and cliclick without requiring browser automation tools.

Can I monitor long-running native applications and dismiss dialogs on macOS?▼

Yes, you can monitor long-running native applications and dismiss dialogs by applying real-time desktop observation with vision models to understand on-screen state and trigger scripted native interactions.

What's the best way to read screen content and detect UI elements on macOS?▼

The best way to read screen content and detect UI elements on macOS is combining the native screencapture utility with vision models to analyze the current on-screen state and guide subsequent UI automation.

Do I need cliclick and osascript to perform scripted UI actions on macOS?▼

Yes, you need cliclick and osascript to perform scripted UI actions on macOS, as these native CLI utilities and AppleScript frameworks handle the clicking, typing, and window management driven by vision analysis.

Does desktop-vision work with OpenClaw integration for native software automation?▼

Yes, desktop-vision works with OpenClaw integration to access vision models for analysis and actions, enabling automated observation and UI interaction for native software and long-running processes on macOS.

Related Skills

Explore More Software Engineering Skills→
screenpipescreenpipe
20.9k

screenpipe-search

Query local Screenpipe OCR, audio, and UI event data with time filters.

Official
Basic
rohunvorarohunvora
20

arena-cli

Export Are.na blocks, enrich with vision AI, and generate browsable HTML views.

Community
Advanced
xtonextone
3

video-keyframe-analyzer

Extract representative keyframes from videos using OpenCV and Pillow.

Official
Intermediate
web-infra-devweb-infra-dev
286

Desktop Computer Automation

Automate desktop environments on macOS, Windows, and Linux via visual input.

Official
Advanced