skills.rest
reason-machines
reason-machinesOFFICIAL
Explore all skills by this author →
Website𝕏@reasonmachinesGitHub

native-devtools-mcp-automation

Automate desktop apps, browsers, and Android devices via an MCP server with screenshots, OCR, and CDP.

4|Updated May 16, 2026
Search Tags:#ocr#UI testing#cdp#ADB#MCP server#desktop automation#accessibility tree
One-click install
npx skills add https://github.com/reason-machines/devtools-skills --skill native-devtools-mcp-automation-reason-machines
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: native-devtools-mcp-automation
Source: https://github.com/reason-machines/devtools-skills/tree/main/skills/native-devtools-mcp-automation
Command: npx skills add https://github.com/reason-machines/devtools-skills --skill native-devtools-mcp-automation-reason-machines

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires native-devtools-mcp.

What problem does it solve? AI agents normally cannot see or interact with native desktop applications, Electron apps, or Android devices. This Skill provides an MCP server that gives agents screenshots, OCR, accessibility-tree access, input simulation, and Chrome DevTools Protocol control so they can operate real user interfaces across macOS, Windows, and Android. ## Core Features & Use Cases - Visual Automation: Take screenshots, find text with OCR, click and type by coordinates, and locate UI elements with template matching on any app. - AX Dispatch and CDP: Use the macOS Accessibility tree for element-precise native app control without moving the cursor, and CDP for DOM-level automation of Chrome and Electron apps. - Android via ADB: Capture screenshots, find text with uiautomator, tap, swipe, type, and launch apps on connected devices. - Use Case: Launch Chrome with remote debugging, connect over CDP, fill a login form using credentials from environment variables, submit it, and wait for the redirect — all driven by an AI agent. ## Quick Start Ask your AI agent to set up the native-devtools MCP server with npx and take a screenshot of the current screen to find and click a button by its text.

Frequently Asked Questions about native-devtools-mcp-automation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate a desktop application with an AI agent?▼

Install the native-devtools-mcp server and connect it to an MCP client like Claude Desktop or Cursor. The agent can then take screenshots, find text with OCR, and simulate clicks and keystrokes, or use the macOS Accessibility tree for precise element control.

How to automate Chrome or Electron apps with CDP?▼

Launch the app with the --remote-debugging-port flag, then connect with cdp_connect on that port. You can navigate pages, find elements, fill inputs, click buttons, evaluate JavaScript, and manage tabs through DOM-level automation.

Does native-devtools-mcp support Windows and Android?▼

Yes. Windows is supported through UI Automation, screenshots, Windows Media OCR, and input simulation. Android devices are controlled over ADB with screenshots, uiautomator text lookup, taps, swipes, and app management.

Why do clicks fail or screenshots return black on macOS?▼

macOS requires Accessibility permission for input simulation and Screen Recording permission for screenshots. Grant both in System Settings under Privacy & Security, otherwise clicks silently fail and screenshots come back black.

What should I use when OCR cannot find text in an app?▼

When OCR fails due to small or low-contrast text, switch to template matching with load_image and find_image, use AX Dispatch snapshots on macOS native apps, or use CDP element queries for web and Electron content.

Related Skills

Explore More Software Engineering Skills→
leprachuanleprachuan

mac-control

Control macOS devices remotely via SSH, AppleScript, and Playwright.

Community
Intermediate
MathiasPaulenkoMathiasPaulenko

webdriverio

Automate end-to-end web and mobile tests with WebdriverIO.

Community
Advanced
johnlindquistjohnlindquist
28

visual-test

Automate visual UI testing for Script Kit GPUI via named pipe screenshots.

Community
Advanced
deepakanirajdeepakaniraj

ai-testing-agent

Automate software testing lifecycle with test case generation, script creation, and defect reporting.

Community
Advanced