skills.rest
hafley66
hafley66COMMUNITY
Explore all skills by this author →
GitHub

macos-screen-text

Locate visible target words on the macOS screen and report pixel coordinates and bounding boxes.

1|Updated Mar 15, 2026
Search Tags:#ocr#macos#rust#text-extraction#bounding-box#screen-coordinates
One-click install
npx skills add https://github.com/hafley66/claude-research --skill macos-screen-text
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: macos-screen-text
Source: https://github.com/hafley66/claude-research/tree/main/skills/macos-screen-text
Command: npx skills add https://github.com/hafley66/claude-research --skill macos-screen-text

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Find and locate every visible instance of a given word on the macOS screen and return its pixel coordinates and bounding boxes.

Core Features & Use Cases

  • Hybrid approaches leveraging AX accessibility trees and Vision OCR to detect on-screen text with pixel-level bounding boxes.
  • Use cases include accessibility automation, UI testing, and screen data extraction across native apps, browsers, and terminal emulators.
  • Requires frontmost app type routing and graceful fallbacks to maintain accuracy and speed.

Quick Start

Trigger a screen-text search for a target word and return all screen-coordinate bounding boxes for every match.

Frequently Asked Questions about macos-screen-text

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find text on a macOS screen and get its pixel coordinates?▼

To find text on a macOS screen and get pixel coordinates, this Skill locates all visible instances of a target word and reports their exact screen-coordinate bounding boxes. It uses AX accessibility trees, Vision OCR, and Chrome DevTools Protocol depending on the frontmost app type.

Can I extract bounding boxes from text in browsers and terminal emulators on macOS?▼

Yes, you can extract text bounding boxes from browsers and terminal emulators on macOS. The Skill detects the frontmost app type and routes extraction through Chrome DevTools Protocol for browsers or terminal APIs, ensuring accurate screen-coordinate outputs.

What is the best way to automate UI testing by locating on-screen text in macOS native apps?▼

The best way to locate on-screen text for macOS UI testing is using hybrid AX accessibility tree walks and Vision OCR. This approach returns precise pixel coordinates and bounding boxes suitable for automation scripts and screen overlays.

Does macOS screen text extraction work across different types of frontmost applications?▼

macOS screen text extraction works across different frontmost applications by using multi-method routing. It automatically detects the app type and applies the appropriate method, gracefully falling back between native AX walks, Vision OCR, and terminal APIs to maintain accuracy.

Why use Vision OCR for screen text extraction instead of just accessibility trees?▼

Vision OCR is used for screen text extraction because accessibility trees do not always expose all visible text or pixel-level bounding boxes. OCR provides a graceful fallback to detect text within canvases or non-native views, ensuring comprehensive coordinate reporting.

Related Skills

Explore More Software Engineering Skills→
sky770825sky770825

screen-vision

Extract screen text and locate element coordinates using macOS Vision Framework.

Community
Intermediate
smykla-skalskismykla-skalski
3

ocr-finder

Detects and extracts text from image files with bounding boxes and click coordinates using EasyOCR.

Official
Intermediate
shuff57shuff57

embedded-browser

Discover Electron remote debugging ports and control browser targets via Chrome DevTools Protocol.

Community
Advanced
CoachSteffCoachSteff
1

vision

Analyze screen content to describe UI, detect elements, and extract text.

Community
Advanced