skills.rest
sky770825
sky770825COMMUNITY
Explore all skills by this author →
GitHub

screen-vision

Extract screen text and locate element coordinates using macOS Vision Framework.

Updated Feb 3, 2026
Search Tags:#automation#ocr#macos#screen recognition#vision framework#local ocr
One-click install
npx skills add https://github.com/sky770825/openclaw-console-hub --skill screen-vision
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: screen-vision
Source: https://github.com/sky770825/openclaw-console-hub/tree/main/skills/screen-vision
Command: npx skills add https://github.com/sky770825/openclaw-console-hub --skill screen-vision

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires swift, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a zero-token cost solution for understanding and interacting with your macOS screen, enabling automation for applications without APIs.

Core Features & Use Cases

  • Screen Text Recognition (OCR): Extract text from any part of your screen without sending data externally.
  • Element Location: Precisely identify the coordinates of text elements on your screen.
  • Automated Actions: Perform clicks and other interactions at specific screen coordinates.
  • Use Case: Automate repetitive tasks in applications like Slack or a custom internal tool by having the AI "see" the screen, locate buttons or information, and click or type as needed.

Quick Start

Use the screen-vision skill to find the text "Submit" on the screen and then click on it.

Frequently Asked Questions about screen-vision

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract text from my macOS screen locally without external APIs?▼

You can extract text from your macOS screen locally by using the Vision Framework, which provides zero-token cost OCR without sending data externally. This approach supports multi-language recognition and precise element coordinate localization.

How do I automate clicks on UI elements in macOS applications without APIs?▼

You can automate interactions in applications without APIs by using the Vision Framework to locate specific text elements on your screen, then performing clicks or typing at those precise coordinates.

Do I need Swift and specific permissions to run macOS screen automation?▼

Yes, executing macOS screen recognition and automation requires a Swift execution environment and specific system permissions for screen recording and accessibility to allow the Vision Framework to interact with your screen.

Can I use the Vision Framework to locate exact coordinates of text on my screen?▼

Yes, you can use the Vision Framework to precisely identify the coordinates of text elements on your screen. This allows you to locate buttons or information and perform automated clicks or typing at those specific screen positions.

What are the limitations of using local OCR for macOS application automation?▼

macOS local OCR automation is limited to recognizing visible screen text and interacting at specific coordinates. It requires specific system permissions for screen recording and accessibility, and cannot interact with hidden UI elements or native accessibility trees.

Related Skills

Explore More Software Engineering Skills→
Zentin-LZentin-L
1

peekaboo

Automate macOS UI tasks by capturing screens, targeting elements, and driving input via CLI.

Community
Advanced
hatayamahatayama
500

simulate-mouse-demo

Simulate mouse clicks, drags, and joystick motions in Unity PlayMode.

Community
Intermediate
cookeyholdercookeyholder

PDF Processing Pro

Automate PDF form analysis, table extraction, OCR, and validation workflows.

Community
Advanced
steipetesteipete
385.7k

peekaboo

Capture and control macOS UI elements via CLI commands.

Community
Advanced