Automate desktop UI clicks and keyboard input using YOLO-detected element IDs.
npx skills add https://github.com/Smyan1909/SoMatic --skill somatic
Please help me install this Agent Skill. Skill: somatic Source: https://github.com/Smyan1909/SoMatic/tree/main Command: npx skills add https://github.com/Smyan1909/SoMatic --skill somatic
SoMatic provides a framework to automate desktop UI tasks by targeting elements with YOLO-detected IDs rather than guessing at pixel coordinates, reducing errors and speeding up automation workflows.
Initialize vision, capture an annotated screenshot, inspect the marks, and click the appropriate mark to operate the UI.
High-intent search queries and answers about installing and using this skill.
You can automate desktop UI tasks by using YOLO-detected element IDs to target buttons and fields. SoMatic annotates GUI surfaces with numbered marks, allowing you to click or type via specific IDs rather than guessing coordinates.
Yes, YOLO object detection can automate native desktop applications and web browsers. SoMatic works across multiple GUI surfaces including native apps, browsers, and PDFs by applying an ONNX model to detect and annotate interactive elements.
MCP-based integration connects agents to a locally running vision daemon that processes GUI screenshots with an ONNX model. The daemon returns annotated marks, which agents then use to perform clicks or keyboard input on the targeted elements.
Yes, a locally running ONNX model is required to capture annotated screenshots for UI automation. The vision daemon loads the ONNX model to detect elements, annotate them with numbered IDs, and enable the CLI operating loop.
The best way to click or type into native apps using detected marks is through mark IDs, offsets, or direct coordinates. SoMatic lets you initialize the vision daemon, inspect the annotated screenshots, and execute actions based on the numbered element IDs.
Coordinate-based UI automation fails because it guesses pixel locations without understanding the actual elements. SoMatic solves this by using YOLO-detected element IDs, which reduces errors and speeds up workflows across changing GUI surfaces.
devMoezDrive macOS desktop interactions via screenshots and AX element indexes.
hungthinh04Control macOS desktop apps in the background without stealing focus.
gaaschkAutomate headless browser actions for QA testing and site dogfooding.
zenobi-usDeploy scalable ML models with real-time inference, monitoring, and CI/CD pipelines.
