skills.rest
gatovillano
gatovillanoCOMMUNITY
Explore all skills by this author →
GitHub

pc_interaction

Automate desktop GUI actions like mouse movement, typing, and screenshots.

5|Updated Aug 9, 2025
Search Tags:#automation#screenshot#windows#keyboard#mouse#automation-tool#gui-automation
One-click install
npx skills add https://github.com/gatovillano/KogniTerm --skill pc-interaction
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pc_interaction
Source: https://github.com/gatovillano/KogniTerm/tree/main/kogniterm/skills/bundled/pc_interaction
Command: npx skills add https://github.com/gatovillano/KogniTerm --skill pc-interaction

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyautogui, pywinctl, and includes scripts (resource) components.

What problem does it solve?

Automates desktop GUI actions, empowering you to move the mouse, type text, manage windows, and capture screenshots without repetitive manual steps.

Core Features & Use Cases

  • get_windows: List open windows to identify targets for automation.
  • activate_window: Focus a specific window by title before interacting.
  • move_mouse, click, double_click, right_click, drag_mouse: Interact with UI elements precisely.
  • type_text, press_key, key_combo: Enter text and keyboard shortcuts.
  • scroll: Scroll within applications.
  • screenshot: Capture the current screen state for verification or documentation.
  • Use Case: Automate a login scenario by bringing the browser window to focus, entering credentials, and taking a screenshot of the result.

Quick Start

Run pc_interaction with action 'get_windows' to list open windows and identify targets for automation.

Frequently Asked Questions about pc_interaction

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate desktop GUI actions like mouse clicks and keyboard typing?▼

Automating desktop GUI actions like mouse clicks and typing is done by controlling the cursor and keyboard input. This tool enables you to move the mouse, type text, manage windows, and capture screenshots without repetitive manual steps.

Can I use pyautogui to automate interactions on both Windows and Linux?▼

Yes, pyautogui works in graphical environments across Windows and Linux. You must ensure a graphical environment is active, requiring a native Windows display or a configured DISPLAY variable on Linux to run automation tasks.

What is the best way to identify and focus a specific window before automating a login?▼

The best way to focus a specific window before automating a login is using the activate_window action. You can first run get_windows to list open windows and identify targets, then focus the browser window to enter credentials.

How do I capture a screenshot to verify the result of a GUI automation task?▼

To capture a screenshot for verification of a GUI automation task, use the screenshot action. This captures the current screen state after completing interactions like moving the cursor or entering text for documentation purposes.

Does GUI automation work in a headless server environment without a display?▼

No, GUI automation does not work in a headless server environment without a display. The tool requires an active graphical environment, meaning it needs native Windows or a configured DISPLAY on Linux to execute mouse and keyboard actions.

Related Skills

Explore More Software Engineering Skills→
wangyendtwangyendt
8

pywayne-gui

Register global hotkeys and manipulate windows on Windows OS.

Community
Intermediate
openaeonopenaeon
87

peekaboo

Automate macOS desktop UI capture and interaction via the Peekaboo CLI.

Community
Advanced
sky770825sky770825

desktop-control

Automate mouse, keyboard, screen capture, and window management on Windows desktops.

Community
Intermediate
EnztunEnztun

Agent Browser

Automate headless browser interactions and extract data from web pages.

Community
Intermediate