skills.rest
jamsdoescode
jamsdoescodeCOMMUNITY
Explore all skills by this author →
GitHub

openclicky-screen-control

Controls OpenClicky's local overlay bridge to point, caption, highlight, and speak on screen.

Updated Jul 7, 2026
Search Tags:#macos#overlay#screen control#http bridge#visual guidance#openclicky#cursor pointing
One-click install
npx skills add https://github.com/jamsdoescode/Erik-s-Portfolio --skill openclicky-screen-control-jamsdoescode
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: openclicky-screen-control
Source: https://github.com/jamsdoescode/Erik-s-Portfolio/tree/main/openclicky/AppResources/OpenClicky/OpenClickyBundledSkills/openclicky-screen-control
Command: npx skills add https://github.com/jamsdoescode/Erik-s-Portfolio --skill openclicky-screen-control-jamsdoescode

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? When users ask "show me where" or "how do I do this", text-only answers force them to hunt for UI elements themselves. This Skill drives OpenClicky's local HTTP bridge to point at on-screen targets, display captions, and speak instructions directly. ## Core Features & Use Cases - On-Screen Pointing: Move OpenClicky's primary cursor to any macOS screen coordinate with a short caption, or show multiple temporary cursors to compare options. - Visual Annotations: Draw freehand scribbles, rectangle highlights, and floating captions near relevant UI elements. - Screenshot-Grounded Targeting: Capture screenshots through the bridge to locate targets before pointing, using AppKit bottom-left-origin coordinates. - Voice Output: Speak guidance through OpenClicky's voice without entering voice mode, with optional interrupt control. - Use Case: A user asks "how do I change this setting?" — capture a screenshot, locate the setting, point at it with a caption like "Click here", and speak a one-line instruction. ## Quick Start Ask the assistant to show you where a specific button or setting is on your screen so it points at the target with a caption through OpenClicky.

Frequently Asked Questions about openclicky-screen-control

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I point at something on screen with OpenClicky?▼

Send a POST request to http://127.0.0.1:32123/cursor with JSON containing x, y coordinates and an optional caption. Coordinates use the macOS AppKit system with the origin at the bottom-left of the desktop.

How do I find the coordinates of a UI element before pointing?▼

Call the /screenshot endpoint on the bridge, which returns local JPEG paths and display frame metadata in the same AppKit coordinate space. Inspect the screenshot to locate the target, then call /cursor with those coordinates.

Does OpenClicky screen control move the real macOS mouse pointer?▼

No, the default /cursor command moves OpenClicky's own triangle cursor overlay, not the real system pointer. A secondary mode exists for extra temporary colored pointers that disappear automatically.

Can OpenClicky speak instructions without entering voice mode?▼

Yes, POST to the /speak endpoint with a text field to have OpenClicky speak without starting dictation or voice mode. If it is already speaking, the call returns HTTP 409 unless you pass interrupt: true.

What visual annotations does the OpenClicky bridge support?▼

The bridge supports cursor pointing, multiple temporary cursors, captions, freehand scribbles via /scribble, and rectangle highlights via /highlight. Spotlight masks, arrows, and persistent annotations are not implemented.

Why is the OpenClicky bridge not responding to commands?▼

Check the bridge health with a GET request to http://127.0.0.1:32123/health or list available tools via /mcp/tools. Some endpoints like /scribble and /highlight require a bearer token in the Authorization header.

Related Skills

Explore More Software Engineering Skills→
moltis-orgmoltis-org
2.8k

peekaboo

Automates macOS UI tasks including capture, inspect, target, and drive inputs via CLI scripting.

Official
Advanced
vercel-labsvercel-labs
904

webreel

Create scripted browser demo videos from JSON step configs.

Official
Advanced
Probably-GroupProbably-Group
3

macos-accessibility

Inspect and control macOS UI elements via AXUIElement with permission checks.

Official
Advanced
autonomy-cloudautonomy-cloud
2

macos-automation

Automate native macOS app interactions using AppleScript via osascript.

Official
Intermediate