android-cli-ui-automation-skill

Automate Android UI interactions via adb and screen capture.

10|Updated Apr 17, 2026

Search Tags:#ui automation #android #agent skill #adb input #layout tree #screenshot targeting #device orchestration

Ready to run

npx skills add https://github.com/takahirom/android-cli-ui-automation-skill --skill android-cli-ui-automation-skill

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: android-cli-ui-automation-skill
Source: https://github.com/takahirom/android-cli-ui-automation-skill/tree/main
Command: npx skills add https://github.com/takahirom/android-cli-ui-automation-skill --skill android-cli-ui-automation-skill

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill removes the guesswork and manual steps of interacting with an Android app by letting an AI repeatedly locate UI elements and perform the corresponding taps, text input, and navigation until your goal is achieved.

Core Features & Use Cases

Launch and navigate apps or URLs: Start installed apps, open a URL via Android intents, or run an APK on a connected device.
Observe UI reliably: Use structured UI trees via android layout and fall back to screenshot labeling with android screen capture/resolve when layout is ambiguous.
Act on the device: Perform taps, text entry, swipes, and key events via adb shell input, re-observing after each action to stay in sync with the screen state.
Multi-device support: Target a specific device/emulator using adb serial (-s) and Android CLI --device.

Quick Start

Ask your agent to launch your APK and tap the “Sign in” button, and the agent will iterate through launch, observe, act, and re-observe steps until the flow completes.

Frequently Asked Questions about android-cli-ui-automation-skill

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate Android UI interactions like tapping and text entry with an AI agent?▼

Automate Android UI interactions by having an AI agent locate elements via layout trees or screenshots, then execute taps, text input, and swipes using adb shell input. The agent re-observes the screen after each action to ensure it reaches your goal.

How does an AI agent locate UI elements on an Android device when the layout tree is ambiguous?▼

When the Android layout tree is ambiguous, the AI agent falls back to screenshot labeling using screen capture and resolve capabilities. This allows it to visually identify and target UI elements that lack clear text, content-desc, or resource-id attributes.

Do I need adb to run Android UI automation flows with an AI agent?▼

Yes, you need adb installed and a connected Android device or emulator. The automation relies on adb shell input for executing actions like taps and swipes, and uses Android CLI for screen capture and layout observation.

Can I target a specific Android emulator or device when automating UI flows?▼

Yes, you can target a specific Android device or emulator during UI automation. The Skill supports multi-device orchestration by allowing you to specify an adb serial number and use the Android CLI device flag.

What is the best way to automate opening an app and navigating a sign-in flow on Android?▼

The best way is to instruct your AI agent to launch the app and tap the sign-in button. The agent iterates through launch, observe, act, and re-observe steps, automatically handling text entry and navigation until the flow completes.

Why does my Android UI automation fail to find the right screen element after tapping?▼

Android UI automation can fail if the agent does not re-observe the screen state after an action. This Skill maintains correctness by re-observing the screen using layout trees or screen capture after every tap or text entry to stay in sync.