app-control

You can automate macOS apps with empty Accessibility trees by driving them through raw keyboard and mouse input, bypassing the AX tree entirely. This approach uses window-relative coordinates to interact directly with custom-rendered interfaces.

Yes, you can control emulators, games, and OpenGL canvases on macOS using raw input. Because these environments render custom UI without standard elements, raw keyboard and mouse actions interact directly with the canvas.

Automating custom-rendered Electron apps requires using raw input proxy tools that observe the window state and then execute keyboard, mouse, and screenshot actions. This ensures deterministic interaction when standard UI navigation fails.

To start a raw input automation session, identify the target app by its bundle ID and initiate the proxy tools. You must observe the window state and allow settle timing before executing input actions to ensure reliable automation.

Raw input automation requires observe and settle timing because it bypasses the Accessibility tree and targets window-relative coordinates. Observing the state before acting ensures sequences, combos, and text input execute reliably without standard UI event feedback.

You should switch to raw input when Accessibility-based automation is unreliable, specifically for emulators, games, OpenGL canvases, and custom-rendered Electron apps. If the AX tree is empty or unhelpful, raw keyboard and mouse input is the necessary alternative.