operate-ui-by-screenshot

What problem does it solve?

This Skill provides a cautious and evidence-driven approach to operating desktop apps and websites, ensuring operational details are explicit for future GUI/web agents.

Core Features & Use Cases

• Structured Browser Interfaces: Utilizes APIs, CLIs, browser DOM inspection, HTML fetch, Playwright-style locators, CDP, or UI Automation for operation.
• Screenshot-Based UI Control: Used when no richer interface is available, with confirmations of visible state before clicking.
• Clipboard Paste: Preferred for long prompts, paths, Chinese text, mixed-language content, and multiline values.
• Verification Screenshots: Captured after clicks, paste actions, navigation, scrolling, exports, and other meaningful actions.
• Profile-Level Self-Optimization: Reuses known app/site profiles and summarizes durable lessons after exploring a new app or website.
• Use Case: Control or inspect a GUI/web app, click buttons, open software, fill forms, scrape/capture HTML, test frontend UI, use Edge/ChatGPT/GitHub/Kaggle/Luogu/Premiere, document UI process screenshots, or summarize reusable lessons from a newly explored app/site where no direct API is enough.

Quick Start

Use the $operate-ui-by-screenshot skill to open a website in Edge, inspect the page, click the visible export button, and save verification screenshots.