record-and-replay

Records browser and desktop user demonstrations and converts them into reusable agent skills.

38|4|Updated Jul 7, 2026
One-click install
npx skills add https://github.com/ugarchance/record-and-replay-skill --skill record-and-replay-ugarchance
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: record-and-replay
Source: https://github.com/ugarchance/record-and-replay-skill
Command: npx skills add https://github.com/ugarchance/record-and-replay-skill --skill record-and-replay-ugarchance

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires playwright, pynput, mss, pillow, and includes scripts (resource) components.

What problem does it solve? Teaching an AI agent a repetitive workflow normally requires writing detailed instructions by hand. This Skill lets you simply demonstrate the task once — in a browser or in native desktop apps — while it records your actions, then turns that recording into a reusable agent skill with semantic locators and verification steps. ## Core Features & Use Cases - Browser recording via Playwright: Captures every user action into events.jsonl with multiple selector candidates (testId, role, id, text, css) plus trace chunks with DOM snapshots, on macOS, Windows, and Linux. - Desktop recording via pynput: Captures global mouse/keyboard events, active-window changes, click-moment screenshots, and macOS accessibility descriptions of clicked elements at roughly 0-3% CPU. - Skill generation from evidence: Summarizers convert recordings into compact markdown timelines that the agent uses to generate a new SKILL.md with semantic Playwright locators or accessibility-tree targets. - Use Case: Demonstrate filling out a weekly report in a web app once; the agent records the session and produces a reusable skill that replays the workflow with parameterized dates and values. ## Quick Start Tell your agent to watch you perform a task and turn the demonstration into a reusable skill, then start the recorder with a named session.

Frequently Asked Questions about record-and-replay

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I record a browser workflow and turn it into an agent skill?

Start the recorder with node scripts/recorder.mjs --minutes 30 --name your-task, perform the workflow in the headed browser, then stop it. Run node scripts/summarize.mjs on the output directory and the agent generates a reusable skill from the recorded events.

How do I record native desktop app actions for an AI agent?

Use node scripts/desktop-record.mjs start your-task 30 to capture mouse, keyboard, window changes, and click screenshots via pynput. Pause, resume, and stop with the same driver, then run scripts/desktop-summarize.py to produce a markdown timeline for skill generation.

Does the desktop recorder work on Windows and Linux?

Yes, desktop recording works on macOS, Windows, and Linux with an X11 session. macOS additionally requires Input Monitoring and Screen Recording permissions granted to the hosting app, which the recorder preflights and reports.

Why is my macOS desktop recording empty?

Empty recordings mean missing macOS permissions: Input Monitoring and Screen Recording must be granted to the app hosting your agent, not to Python, and that app must be restarted. Run node scripts/desktop-record.mjs permissions to check and request them.

Are passwords and sensitive data masked in recordings?

Browser events.jsonl masks password, OTP, and card fields heuristically, but trace chunks are not masked. Desktop recordings have no masking at all, so treat the entire recording directory as sensitive and never commit or share it.

What are the limitations of replaying recorded workflows?

Generated skills use semantic Playwright locators or accessibility-tree targets rather than raw coordinates, but desktop events are weaker than browser selectors. Prefer API or CLI connectors over UI replay when they achieve the same outcome.