desktop-automation

Automate NixOS Hyprland desktop interactions including window control, input, screenshots, and Chromium CDP.

1|1|Updated Nov 25, 2024
One-click install
npx skills add https://github.com/joeledwardson/dev-setup --skill desktop-automation-joeledwardson
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: desktop-automation
Source: https://github.com/joeledwardson/dev-setup/tree/main/configs/claude/skills/desktop-automation
Command: npx skills add https://github.com/joeledwardson/dev-setup --skill desktop-automation-joeledwardson

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Tasks that require a real graphical UI — testing click flows, verifying visual output, pasting into a running TUI — cannot be done headlessly. This Skill provides the tools and patterns to drive a NixOS + Hyprland desktop programmatically without human input. ## Core Features & Use Cases - Window and Input Control: Send key chords to specific windows via hyprctl, type text with wtype, and click with ydotool, following an observe-act-verify loop. - Screenshots and Clipboard: Capture region screenshots with grim to stay under context limits, and manage the clipboard with wl-copy/wl-paste. - Browser Automation via CDP: Drive Chromium/Brave through the Chrome DevTools Protocol for DOM-level navigation, JavaScript evaluation, and clipped screenshots without focus handling. - Use Case: While SSH'd into a NixOS machine, export the Wayland session variables, launch Brave with a remote debugging port, navigate to a local web app, and verify the rendered page with a clipped CDP screenshot. ## Quick Start Set the Wayland and Hyprland session environment variables, then use the desktop-automation skill to screenshot the active window and verify its state.

Frequently Asked Questions about desktop-automation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate keyboard input on Hyprland from a script?

Use hyprctl dispatch with hl.dsp.send_shortcut to send key chords to a specific window class, which bypasses focus requirements. For free-form text to the focused window, use wtype, for example wtype -M ctrl -p v -m ctrl for Ctrl+V.

How to automate Chromium browser testing without screenshots and clicks?

Launch the browser with --remote-debugging-port and a fresh --user-data-dir, then drive it via the Chrome DevTools Protocol. Use Page.navigate, Runtime.evaluate for DOM interaction, and Page.captureScreenshot with a clip region for visual checks.

Why do hyprctl and grim fail when run over SSH?

SSH sessions lack WAYLAND_DISPLAY and HYPRLAND_INSTANCE_SIGNATURE, so every Wayland tool fails silently. Export both variables from /run/user/$(id -u)/ before running any desktop automation commands.

Why does ydotool fail with permission denied?

ydotool requires the ydotoold daemon running and your user in the ydotool group; check group membership with id. For keyboard-only tasks, fall back to wtype or hyprctl send_shortcut instead.

How do I avoid hitting context limits with screenshots?

Cumulative screenshot dimensions above roughly 2000px cause request rejection, so default to region grabs with grim -g using geometry from hyprctl clients -j. Take one screenshot per step and prefer CDP clipped captures for browser content.

When should I not use desktop automation?

Avoid automating the user's accounts, messaging apps, or anything tied to their identity, and never steal focus while the user is actively working. If three observe-act-verify cycles fail on the same step, stop and re-evaluate the UI model.