android-cli-ui-automation-skill

Automate Android UI interactions by having an AI agent locate elements via layout trees or screenshots, then execute taps, text input, and swipes using adb shell input. The agent re-observes the screen after each action to ensure it reaches your goal.

When the Android layout tree is ambiguous, the AI agent falls back to screenshot labeling using screen capture and resolve capabilities. This allows it to visually identify and target UI elements that lack clear text, content-desc, or resource-id attributes.

Yes, you need adb installed and a connected Android device or emulator. The automation relies on adb shell input for executing actions like taps and swipes, and uses Android CLI for screen capture and layout observation.

Yes, you can target a specific Android device or emulator during UI automation. The Skill supports multi-device orchestration by allowing you to specify an adb serial number and use the Android CLI device flag.

The best way is to instruct your AI agent to launch the app and tap the sign-in button. The agent iterates through launch, observe, act, and re-observe steps, automatically handling text entry and navigation until the flow completes.

Android UI automation can fail if the agent does not re-observe the screen state after an action. This Skill maintains correctness by re-observing the screen using layout trees or screen capture after every tap or text entry to stay in sync.