screen-vision

Name: screen-vision
Availability: InStock
Author: sky770825

Extract screen text and locate element coordinates using macOS Vision Framework.

Updated Feb 3, 2026

Search Tags:#automation #ocr #macos #screen recognition #vision framework #local ocr

Ready to run

npx skills add https://github.com/sky770825/openclaw-console-hub --skill screen-vision

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: screen-vision
Source: https://github.com/sky770825/openclaw-console-hub/tree/main/skills/screen-vision
Command: npx skills add https://github.com/sky770825/openclaw-console-hub --skill screen-vision

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires swift, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a zero-token cost solution for understanding and interacting with your macOS screen, enabling automation for applications without APIs.

Core Features & Use Cases

Screen Text Recognition (OCR): Extract text from any part of your screen without sending data externally.
Element Location: Precisely identify the coordinates of text elements on your screen.
Automated Actions: Perform clicks and other interactions at specific screen coordinates.
Use Case: Automate repetitive tasks in applications like Slack or a custom internal tool by having the AI "see" the screen, locate buttons or information, and click or type as needed.

Quick Start

Use the screen-vision skill to find the text "Submit" on the screen and then click on it.

Frequently Asked Questions about screen-vision

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extract text from my macOS screen locally without external APIs?▼

You can extract text from your macOS screen locally by using the Vision Framework, which provides zero-token cost OCR without sending data externally. This approach supports multi-language recognition and precise element coordinate localization.

How do I automate clicks on UI elements in macOS applications without APIs?▼

You can automate interactions in applications without APIs by using the Vision Framework to locate specific text elements on your screen, then performing clicks or typing at those precise coordinates.

Do I need Swift and specific permissions to run macOS screen automation?▼

Yes, executing macOS screen recognition and automation requires a Swift execution environment and specific system permissions for screen recording and accessibility to allow the Vision Framework to interact with your screen.

Can I use the Vision Framework to locate exact coordinates of text on my screen?▼

Yes, you can use the Vision Framework to precisely identify the coordinates of text elements on your screen. This allows you to locate buttons or information and perform automated clicks or typing at those specific screen positions.

What are the limitations of using local OCR for macOS application automation?▼

macOS local OCR automation is limited to recognizing visible screen text and interacting at specific coordinates. It requires specific system permissions for screen recording and accessibility, and cannot interact with hidden UI elements or native accessibility trees.