Imported from rishi058/REDROID (skills/ui-automator/SKILL.md). Install upstream with npx skills add rishi058/REDROID --skill ui-automator. Copyright stays with the author.

Android UI Automation via adb + uiautomator

Purpose

Drive an Android UI without human fingers: read what is on screen, compute where to tap, act, and verify the result. Works over plain adb (SSH tunnel, USB, TCP). No Appium/instrumentation needed — only adb, Python 3 (stdlib), and bash.

Core mental model: the DUMP-TAP-VERIFY loop

Never tap blind. Every interaction follows this loop:

1. STATE   confirm foreground app/window (dumpsys window)
2. DUMP    uiautomator dump /sdcard/x.xml → adb pull
3. PARSE   python scripts/parse_ui.py x.xml
           (or find_node.py for a specific label)
4. LOCATE  bounds="[x1,y1][x2,y2]" → center = ((x1+x2)/2, (y1+y2)/2)
5. ACT     adb shell input tap X Y  (or swipe/keyevent/text)
6. VERIFY  sleep 2-5 s → fresh dump → assert new state

Golden rules

1. Coordinates are disposable; the loop is not. Never reuse coordinates from a previous dump — layouts shift. Always re-dump before every tap.
2. Match text/content-desc first, coordinates last.
3. Verify after every action with a fresh dump or mCurrentFocus.
4. One device per command: always -s SERIAL.

Setup / prerequisites

# 1. Tunnel (if device lives behind SSH) — run as background job:
ssh -i <key> -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
    -N -L 127.0.0.1:5557:127.0.0.1:5557 user@host

# 2. Connect + sanity check
adb connect 127.0.0.1:5557 && adb devices       # expect: ...device
adb -s 127.0.0.1:5557 shell getprop sys.boot_completed   # must print "1"

Command reference (quick)

Full details: references/adb-uiautomator-cheatsheet.md

Scripts (in scripts/)

Worked examples (real sequences from this repo)

A. Open Play Store account sheet → Play Protect (scrollable bottom sheet)

SER=127.0.0.1:5557
adb -s $SER shell input tap 648 104                    # avatar (found via list_top_nodes.py)
sleep 3
adb -s $SER shell input swipe 360 1000 360 450 400     # scroll; rows ~116 px apart
bash ui_dump.sh $SER sheet.xml
python find_node.py sheet.xml "Play Protect"           # → CENTER=(360, 898)
adb -s $SER shell input tap 360 898                    # tap row center, not text edge

B. Capture a Play Integrity verdict

adb -s $SER shell am force-stop gr.nikolasspyr.integritycheck
adb -s $SER shell am start -W -n gr.nikolasspyr.integritycheck/.MainActivity
adb -s $SER shell input tap 360 877        # CHECK button ([272,829][448,925])
sleep 20                                   # token round-trip takes 10-18 s
bash ui_dump.sh $SER v.xml && python check_verdict.py v.xml

Note: the checker's raw-JSON dialog body is NOT visible in uiautomator dumps and clipboard reads are blocked on Android 14 — read verdicts from the Pass/Fail icons.

More recipes: references/proven-workflows.md

Decision tree

Need to interact with Android UI
├─ Device connected?          no → tunnel up → adb connect → boot_completed=1
├─ Target app foreground?     no → am start -W ... (verify Status: ok)
├─ Element has text?          yes → dump → find_node.py "<text>" → tap CENTER
│    └─ below fold?                → input swipe up → re-dump → retry
├─ Element is an icon?
│   ├─ content-desc present?      → find_node.py --desc "<desc>"
│   └─ unlabeled header item?     → list_top_nodes.py, identify by class+bounds
├─ Need action result?        sleep → fresh dump → parse/assert state
└─ Something misbehaved?      references/troubleshooting.md

Safety notes

• pm clear PKG wipes ALL app data (GSF ids, sign-ins, scan history) — only use with explicit intent.
• Do not spam taps against rate-limited flows (integrity checks, logins).
• On shared hosts: container restarts usually fine; full VPS reboots may be forbidden — check project docs first.

Host pitfalls digest (Windows/PowerShell)

1. No < stdin redirect → pipe files: Get-Content -Raw s.sh | ssh h "tr -d '\r' | bash -s"
2. & starts PS background jobs; chain commands with ;.
3. $C:/path inside double quotes breaks PS variable parsing — avoid $var: patterns.
4. Nested-quote python one-liners are fragile — use script files from scripts/.
5. Strip CRLF (tr -d '\r') before piping scripts to remote bash.

Full symptom/fix table: references/troubleshooting.md