Extract click coordinates from screenshots and click non-DOM elements.
npx skills add https://github.com/lidge-jun/cli-jaw-skills --skill vision-click
Please help me install this Agent Skill. Skill: vision-click Source: https://github.com/lidge-jun/cli-jaw-skills/tree/main/vision-click Command: npx skills add https://github.com/lidge-jun/cli-jaw-skills --skill vision-click
This Skill enables interaction with non-DOM elements on a webpage by using AI to interpret visual information and determine click coordinates, overcoming limitations of traditional element-based clicking.
Use the vision-click skill to click the "Submit button" on the current screen.
High-intent search queries and answers about installing and using this skill.
To click elements inside a canvas or Shadow DOM, use AI vision-based coordinate extraction to analyze screenshots and determine visual coordinates for mouse interaction. This bypasses standard DOM selectors by interpreting pixels, enabling clicks on dynamically rendered or complex UI elements.
Yes, you need Codex CLI to perform vision-based clicking, as it handles the AI analysis of screenshots for coordinate extraction. You also need cli-jaw to manage browser interaction and capture the screenshots required for processing.
The best way to automate clicks on dynamically rendered content without HTML IDs is using AI vision analysis to extract visual coordinates from screenshots. This approach interprets visual appearance rather than DOM structure, successfully targeting elements in iframes or canvas.
Yes, AI vision analysis can identify click coordinates for buttons within complex web applications by evaluating their visual appearance in a screenshot. It processes the image to locate the element and calculates precise coordinates for mouse execution.
Browser automation scripts fail on iframes and Shadow DOM because standard element selectors cannot penetrate these boundaries. Vision-based coordinate extraction solves this by analyzing a screenshot visually and clicking the target coordinates directly.
To click a Submit button on a canvas element, invoke the vision-click skill to capture a screenshot, use Codex CLI for AI visual analysis to extract the button coordinates, and execute the mouse click via cli-jaw.
lidge-junAutomate Google Chrome browser interactions for web automation tasks.
YanBerdinDrive a browser to navigate, interact with elements, and extract data.
atomeamCapture screenshots, inspect DOM elements, and test selectors in browser automation.
X-School-AcademyIdentify UI elements in screenshots and generate bounding box coordinates.
