Automate mouse, keyboard, and screen reading on macOS via pyautogui and AppleScript.
npx skills add https://github.com/dalehurley/phpbot --skill desktop-control
Please help me install this Agent Skill. Skill: desktop-control Source: https://github.com/dalehurley/phpbot/tree/main/skills/desktop-control Command: npx skills add https://github.com/dalehurley/phpbot --skill desktop-control
Automates desktop interactions to replace repetitive manual GUI work by controlling mouse, keyboard, and screen reading.
Run a sample script to move the mouse to a target position, click, and type text to automate a simple workflow.
High-intent search queries and answers about installing and using this skill.
You can automate mouse and keyboard inputs on macOS using bundled Python scripts based on pyautogui. The scripts control mouse movements, clicks, scrolling, and keystrokes to replace repetitive manual GUI work.
Yes, you can read the macOS accessibility tree to extract UI state and guide automated actions. This allows the workflow to identify controls and screen content before performing clicks or typing text.
Yes, macOS requires accessibility permissions to allow automated desktop control. The CLI workflow includes guardrails and permission checks to ensure scripts can properly access the UI and execute pyautogui actions.
The best way to automatically fill a login form is by reading the UI accessibility tree to locate fields, then using keyboard control to type text and execute required keystrokes for submission.
Pyautogui handles mouse and keyboard control, while the scripts also leverage AppleScript UI access to read the accessibility tree. This combination enables screen reading and UI state extraction to guide workflows.
Limitations include requiring explicit accessibility permissions and relying on the stability of the UI layout. If interface elements shift unexpectedly, the pyautogui-based mouse and keyboard coordinates may fail.
lubancafeValidate sandboxed skill loading and script execution in Node.js environments.
dawsonblockAutomate mouse and keyboard interactions with screen capture for GUI testing.
PPPPandaControls mouse, keyboard, and screen capture for GUI workflows via pyautogui-based actions with optional OpenCV template matching.
mikeyobrienDisplay native macOS notifications with title, message, and optional sound.
