diff --git a/docs/roadmap/maestro-competitive-roadmap-COMPLETED.md b/docs/roadmap/maestro-competitive-roadmap-COMPLETED.md new file mode 100644 index 00000000..2f6f7105 --- /dev/null +++ b/docs/roadmap/maestro-competitive-roadmap-COMPLETED.md @@ -0,0 +1,68 @@ +# Maestro-competitive roadmap — completion report + +**Status: all 19 items complete** (2026-08-15). This is the consolidated summary; per-item detail, +caveats, and honest limitations live in each item's entry in +[`maestro-competitive-roadmap.md`](maestro-competitive-roadmap.md) and its linked +`docs/evidence/` folder. Shipped across releases **v0.11.0 → v0.13.0**. + +## Phase 0 — Correctness (shipped in v0.11.0) + +| Item | What shipped | +|---|---| +| R-1 | Tap-family verbs now resolve `` placeholders in their selectors (previously only `type`/`see` did) | +| R-3 / PT-27 | Android reconnection token survives OS cache clearing — the periodic re-print now also re-writes the token file | +| R-4 | `see #field contains "…"` reads `EditableText` controllers, not just `Text`/`RichText` | +| V-1 / PT-03 | Scroll targeting re-verified holding on real devices (stacked-route Scrollables) | +| R-5 | `dump tree` / `save device logs` parse correctly (dangling-token misparse fixed) | + +## Phase 0.5 — Docs + benchmark groundwork (v0.11.0) + +| Item | What shipped | +|---|---| +| D-1 | `dictionary.md` completeness pass — all missing verb rows added | +| D-2 | `ai:` config block documented in `configuration.md` (provider/api_key/model/endpoint/timeout/redact) | +| B-1 | water-sip's 9-flow suite written twice — ProbeScript and equivalent Maestro YAML — as the benchmark corpus | +| B-2 | Reusable benchmark harness: `scripts/bench/run-comparison.sh` + `summarize.py` (later gained `--order` and `--android-adb-port` in G-1) | +| B-3 | N=10 baseline: FlutterProbe 59.6s median / 0% flake vs Maestro 122.6s / 100% flake (~2.1×); device-connectivity finding flagged for replication, later retired in G-1 | + +## Phase 1 — Ecosystem parity (v0.12.0, PRs #227–#231) + +| Item | What shipped | +|---|---| +| E-1 (#228) | `retry N times` block (re-run-on-failure, stop at first success) + trailing `optional` modifier on tap/type/long press/double tap/clear/see | +| E-2 (#229) | Element-scoped visual regression: `compare screenshot "x" of ` crops both baseline and actual to the widget's bounds | +| E-3 (#230) | Deep links: `open link "…" in the app` via OS intent handling (`am start -a VIEW` / `simctl openurl`). Documented caveat: iOS Simulator cannot cold-launch a terminated app | +| E-4 (#231) | `add media "path"` seeds the camera roll/gallery (`adb push` + media-scan broadcast / `simctl addmedia`), MediaStore-indexed and picker-visible | + +## Phase 2 — Native UI bridging (v0.12.0, PRs #232–#233) + +| Item | What shipped | +|---|---| +| N-1 (#232) | `tap native` / `see native` / `type native` on Android via uiautomator — pickers, share sheets, anything outside the Flutter tree. Zero new dependencies. Dedicated fixture app added later in #241 | +| N-2 (#233) | iOS native bridging proposal (`docs/proposals/n2-ios-native-ui-bridging.md`): WebDriverAgent, Simulator-first, reusing N-1's syntax unchanged. Proposal only — iOS verbs error clearly by design today. Swift fixture app for the spike shipped in #241 | + +## Phase 3 — Compound the lead (v0.12.0, PRs #234–#236) + +| Item | What shipped | +|---|---| +| G-1 (#234) | Benchmark re-run with current build, **both orderings** (probe-first + maestro-first): ~2.0–2.1× median wall-clock advantage held both ways, 0% flake across 20 FlutterProbe runs. B-3's device-connectivity finding **did not replicate — retired**, said so publicly. Blog post: `/blog/flutterprobe-vs-maestro-benchmark/` | +| G-2 (#235) | vs-Maestro comparison refreshed with 2026 facts: no physical iOS, ~$250/device/mo cloud, AI assertions at parity with a stronger local-first privacy story, native-UI gap narrowed | +| G-3 (#236) | `probe migrate maestro` hardened: 2.x syntax (setPermissions/retry/assertScreenshot + the corpus's real gaps extendedWaitUntil/scrollUntilVisible/eraseText), recursive directory discovery, relativePoint corruption fixed. 76/76 real flows convert and parse | + +## Shipped beyond the roadmap in the same cycle + +- **v0.12.0/v0.12.1 recovery + campaign fixes** (#238, #242): issue-#237 reconnect-retry dispatch + drift, conditional connection-error misrouting, `toggle` double no-op, filler-word recipe names, + Android `set location`, iOS permission relaunch, CLI-side duration waits — eight real bugs found + by dogfooding probe against real apps, all with regression tests and live device verification. +- **adb transport forensics** (#239): host adb server flush-timeout socket closes identified as + the likely #237 drop mechanism, with the raw server log preserved; guest-adbd wedge reproduced + without Maestro. +- **Full-feature campaign** (#243): every ProbeScript feature exercised on both platforms with a + local Gemma 4 model for the AI family — Android 32/32, iOS 31/32 (the "failure" being the AI + correctly catching a real app-layout defect). +- **Native fixture apps** (#241) and the **`migrate_maestro` MCP tool** (#245, v0.13.0) closing + the MCP surface gap. +- **Docs surface sync** (#240, #244, platform-pages PR): dictionary, syntax, visual-regression, + README, landing page (version badge now build-time-generated from `VERSION`), VS Code grammar, + MCP tool descriptions, wiki, and both platform pages. diff --git a/website/src/content/docs/platform/android.md b/website/src/content/docs/platform/android.md index c596bc9e..a324f552 100644 --- a/website/src/content/docs/platform/android.md +++ b/website/src/content/docs/platform/android.md @@ -93,6 +93,59 @@ grant all permissions # grants all known runtime permissions Available permissions: `notifications`, `camera`, `location`, `microphone`, `storage`, `contacts`, `phone`, `calendar`, `sms`, `bluetooth`. +## Native UI Automation + +Android is currently the only platform where probe can reach **outside the Flutter widget tree** +into native, OS-owned UI — pickers, share sheets, and any other surface the Dart agent can never +see. Elements are matched against uiautomator's text or resource-id and driven via +`uiautomator dump` + `input tap`/`input text` — no new dependencies, `adb` already ships both: + +``` +tap native "Choose from Gallery" +see native "IMG_0001.jpg" +don't see native "Error" +type native "wifi" into "Search settings" +``` + +Notes from real-device verification: + +- Matching is a case-insensitive substring against both `text` and `resource-id`. +- If the native element is reached via a screen transition, add a `wait` step first — the same + idiom used after Flutter navigation. +- `take a screenshot` (the Dart-agent verb) cannot capture native UI — it renders only Flutter's + own tree. Use `adb exec-out screencap` externally to visually verify native state. +- API 35 pitfall for your own native screens: forced edge-to-edge can place a top-of-screen + element's reported bounds center underneath the app bar, which silently eats the tap — handle + window insets (`fitsSystemWindows` or equivalent). + +A purpose-built native fixture app with stable ids lives in the repo under `native-test-apps/`. + +## Device Media + +``` +add media "fixtures/photo.jpg" # adb push + MEDIA_SCANNER_SCAN_FILE broadcast +``` + +The file lands in `/sdcard/Pictures/` and is MediaStore-indexed, so it is genuinely visible to +image pickers — combine with the native UI verbs above to actually select it. + +## Deep Links + +``` +open link "myapp://profile/42" in the app +``` + +Dispatches `am start -a android.intent.action.VIEW`, so a custom scheme or App Links URL +registered by your app is delivered to the app itself — including cold-launching it from fully +terminated (unlike iOS Simulator; see the iOS page). Plain `open link "https://..."` (no suffix) +still opens the external browser via `url_launcher`. + +## Location + +``` +set location 48.1351, 11.5820 # adb emu geo fix (emulators only) +``` + ## Video Recording Android uses the built-in `screenrecord` command. Videos are recorded as MP4 (H.264). The CLI auto-chains recordings to work around the 180-second limit. diff --git a/website/src/content/docs/platform/ios.md b/website/src/content/docs/platform/ios.md index 329dccab..e0e56d41 100644 --- a/website/src/content/docs/platform/ios.md +++ b/website/src/content/docs/platform/ios.md @@ -98,6 +98,14 @@ revoke all permissions This works on iOS 14+ simulators. +:::note[Permission changes relaunch the app (v0.12.1+)] +`simctl privacy grant/revoke` silently **terminates the target app** — an Apple behavior, not a +probe one. As of v0.12.1, all four permission verbs automatically relaunch the app and reconnect +the session afterward, so a permission step costs a few seconds of restart instead of hanging the +next step. Follow a permission step with `wait until "" appears` before asserting, +exactly as you would after `restart the app`. +::: + :::caution[Notification permissions cannot be pre-granted] Apple does not support granting notification permissions via `simctl privacy`. If your app requests notification permission (e.g. via `UNUserNotificationCenter` or Firebase Messaging), the native dialog will block the Flutter UI and prevent tests from proceeding. @@ -114,6 +122,41 @@ if (!probeEnabled) { Build with `--dart-define=PROBE_AGENT=true` to skip these requests during testing. ::: +## Device Media + +``` +add media "fixtures/photo.jpg" # xcrun simctl addmedia +``` + +The file lands in the simulator's Photos library and is immediately visible to image pickers. +Simulators only — there is no `devicectl` equivalent for physical devices; the step skips with a +warning there. + +## Deep Links + +``` +open link "myapp://profile/42" in the app +``` + +Dispatches `xcrun simctl openurl`, so a custom scheme or Universal Link registered by your app is +delivered to the app itself. Plain `open link "https://..."` still opens Safari via +`url_launcher`. + +:::caution[No cold launch on iOS Simulator] +`simctl openurl` cannot launch a fully-terminated app — iOS shows an "Open in App?" confirmation +dialog that `simctl` has no way to dismiss. Deep links work reliably when the app is already +running (foreground or backgrounded); put a `restart the app` or equivalent warm-up before the +deep-link step if the app might not be running. Android has no such caveat. +::: + +## Native UI Automation + +The `tap native` / `see native` / `type native` verbs are **Android-only today** — iOS has no +`uiautomator` equivalent, and `simctl` cannot inspect or drive arbitrary UI. On iOS, +`tap native`/`type native` fail with a clear error rather than silently no-op'ing, and +`see native` reports "not found". iOS support via WebDriverAgent is a written, scoped proposal +(`docs/proposals/n2-ios-native-ui-bridging.md` in the repo), not yet implemented. + ## Biometric Authentication (Face ID / Touch ID) `enroll biometric`, `biometric match`, and `biometric no match` drive Face ID and Touch ID flows on iOS Simulator via `xcrun simctl spawn booted notifyutil`.