skills.rest
bdambrosio
bdambrosioCOMMUNITY
Explore all skills by this author →
GitHub

osworld-execute

Execute Python code with pyautogui GUI automation in OSWorld.

10|Updated Jul 22, 2025
Search Tags:#automation#python#gui#execute#pyautogui#osworld
One-click install
npx skills add https://github.com/bdambrosio/Cognitive_workbench --skill osworld-execute
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: osworld-execute
Source: https://github.com/bdambrosio/Cognitive_workbench/tree/main/src/world-tools/osworld/osworld-execute
Command: npx skills add https://github.com/bdambrosio/Cognitive_workbench --skill osworld-execute

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill allows for the execution of arbitrary Python code, particularly GUI automation commands, within the OSWorld environment, enabling interaction with the operating system and applications.

Core Features & Use Cases

  • Python Code Execution: Run Python scripts directly.
  • GUI Automation: Utilize libraries like pyautogui for mouse clicks, typing, and keyboard presses.
  • Observation Capture: Optionally capture the screen state after execution.
  • Use Case: Automate repetitive tasks like filling out web forms, clicking buttons in desktop applications, or generating screenshots.

Quick Start

Use the osworld-execute skill to click the mouse at coordinates 100, 200.

Frequently Asked Questions about osworld-execute

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I execute Python code for GUI automation within an OS environment?▼

You can execute Python code for GUI automation by running scripts that utilize libraries like pyautogui for mouse clicks and keyboard presses to interact with desktop applications.

Can I capture the screen state after running GUI automation commands?▼

Yes, you can optionally capture screen observations after execution to verify the results of your GUI automation commands and check the final state of the operating system.

How do I automate repetitive desktop tasks like filling out web forms?▼

Automate repetitive desktop tasks by executing Python scripts that use pyautogui to simulate mouse clicks, typing, and keyboard presses to interact with application interfaces.

Does executing Python scripts in this environment return standard output and error codes?▼

Executing Python scripts captures execution results including return codes, standard output, and standard error, allowing you to monitor and debug your automation processes.

Do I need a specific server URL to configure the OSWorld environment?▼

You need to configure the OSWorld server URL, which can be set via environment variables or character configuration, to enable the execution of GUI automation commands.

What are the limitations of using pyautogui for OS automation?▼

Using pyautogui for OS automation relies on coordinate-based interactions, which may break if the GUI layout changes or if applications do not maintain consistent window positions.

Related Skills

Explore More Software Engineering Skills→
bdambrosiobdambrosio
10

osworld-status

Retrieve OSWorld server operational status including readiness, uptime, and configuration.

Community
Basic
AstorYHAstorYH
15

python-script-trigger

Execute user-provided Python scripts in a sandboxed environment with timeouts.

Community
Intermediate
ammonfifeammonfife

mac-control

Automate Mac desktop control with Python scripts for windows, mouse, keyboard, and screenshots.

Community
Intermediate
thistleknotthistleknot

cua-desktop-agent

Automate desktop GUI tasks via vision-language perception and CUA-formatted actions.

Community
Advanced