Skip to content

docs: platform-page feature sync + roadmap completion report #246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pbertsch merged 2 commits into from
Aug 16, 2026
Merged

docs: platform-page feature sync + roadmap completion report #246

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d508c9b
docs: sync platform pages with the 0.12.x feature set
pbertsch Aug 16, 2026
61d3e57
docs: roadmap completion report -- all 19 items, phases, versions, PRs
pbertsch Aug 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions docs/roadmap/maestro-competitive-roadmap-COMPLETED.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Maestro-competitive roadmap — completion report

**Status: all 19 items complete** (2026-08-15). This is the consolidated summary; per-item detail,
caveats, and honest limitations live in each item's entry in
[`maestro-competitive-roadmap.md`](maestro-competitive-roadmap.md) and its linked
`docs/evidence/` folder. Shipped across releases **v0.11.0 → v0.13.0**.

## Phase 0 — Correctness (shipped in v0.11.0)

| Item | What shipped |
|---|---|
| R-1 | Tap-family verbs now resolve `<var>` placeholders in their selectors (previously only `type`/`see` did) |
| R-3 / PT-27 | Android reconnection token survives OS cache clearing — the periodic re-print now also re-writes the token file |
| R-4 | `see #field contains "…"` reads `EditableText` controllers, not just `Text`/`RichText` |
| V-1 / PT-03 | Scroll targeting re-verified holding on real devices (stacked-route Scrollables) |
| R-5 | `dump tree` / `save device logs` parse correctly (dangling-token misparse fixed) |

## Phase 0.5 — Docs + benchmark groundwork (v0.11.0)

| Item | What shipped |
|---|---|
| D-1 | `dictionary.md` completeness pass — all missing verb rows added |
| D-2 | `ai:` config block documented in `configuration.md` (provider/api_key/model/endpoint/timeout/redact) |
| B-1 | water-sip's 9-flow suite written twice — ProbeScript and equivalent Maestro YAML — as the benchmark corpus |
| B-2 | Reusable benchmark harness: `scripts/bench/run-comparison.sh` + `summarize.py` (later gained `--order` and `--android-adb-port` in G-1) |
| B-3 | N=10 baseline: FlutterProbe 59.6s median / 0% flake vs Maestro 122.6s / 100% flake (~2.1×); device-connectivity finding flagged for replication, later retired in G-1 |

## Phase 1 — Ecosystem parity (v0.12.0, PRs #227–#231)

| Item | What shipped |
|---|---|
| E-1 (#228) | `retry N times` block (re-run-on-failure, stop at first success) + trailing `optional` modifier on tap/type/long press/double tap/clear/see |
| E-2 (#229) | Element-scoped visual regression: `compare screenshot "x" of <selector>` crops both baseline and actual to the widget's bounds |
| E-3 (#230) | Deep links: `open link "…" in the app` via OS intent handling (`am start -a VIEW` / `simctl openurl`). Documented caveat: iOS Simulator cannot cold-launch a terminated app |
| E-4 (#231) | `add media "path"` seeds the camera roll/gallery (`adb push` + media-scan broadcast / `simctl addmedia`), MediaStore-indexed and picker-visible |

## Phase 2 — Native UI bridging (v0.12.0, PRs #232–#233)

| Item | What shipped |
|---|---|
| N-1 (#232) | `tap native` / `see native` / `type native` on Android via uiautomator — pickers, share sheets, anything outside the Flutter tree. Zero new dependencies. Dedicated fixture app added later in #241 |
| N-2 (#233) | iOS native bridging proposal (`docs/proposals/n2-ios-native-ui-bridging.md`): WebDriverAgent, Simulator-first, reusing N-1's syntax unchanged. Proposal only — iOS verbs error clearly by design today. Swift fixture app for the spike shipped in #241 |

## Phase 3 — Compound the lead (v0.12.0, PRs #234–#236)

| Item | What shipped |
|---|---|
| G-1 (#234) | Benchmark re-run with current build, **both orderings** (probe-first + maestro-first): ~2.0–2.1× median wall-clock advantage held both ways, 0% flake across 20 FlutterProbe runs. B-3's device-connectivity finding **did not replicate — retired**, said so publicly. Blog post: `/blog/flutterprobe-vs-maestro-benchmark/` |
| G-2 (#235) | vs-Maestro comparison refreshed with 2026 facts: no physical iOS, ~$250/device/mo cloud, AI assertions at parity with a stronger local-first privacy story, native-UI gap narrowed |
| G-3 (#236) | `probe migrate maestro` hardened: 2.x syntax (setPermissions/retry/assertScreenshot + the corpus's real gaps extendedWaitUntil/scrollUntilVisible/eraseText), recursive directory discovery, relativePoint corruption fixed. 76/76 real flows convert and parse |

## Shipped beyond the roadmap in the same cycle

- **v0.12.0/v0.12.1 recovery + campaign fixes** (#238, #242): issue-#237 reconnect-retry dispatch
drift, conditional connection-error misrouting, `toggle` double no-op, filler-word recipe names,
Android `set location`, iOS permission relaunch, CLI-side duration waits — eight real bugs found
by dogfooding probe against real apps, all with regression tests and live device verification.
- **adb transport forensics** (#239): host adb server flush-timeout socket closes identified as
the likely #237 drop mechanism, with the raw server log preserved; guest-adbd wedge reproduced
without Maestro.
- **Full-feature campaign** (#243): every ProbeScript feature exercised on both platforms with a
local Gemma 4 model for the AI family — Android 32/32, iOS 31/32 (the "failure" being the AI
correctly catching a real app-layout defect).
- **Native fixture apps** (#241) and the **`migrate_maestro` MCP tool** (#245, v0.13.0) closing
the MCP surface gap.
- **Docs surface sync** (#240, #244, platform-pages PR): dictionary, syntax, visual-regression,
README, landing page (version badge now build-time-generated from `VERSION`), VS Code grammar,
MCP tool descriptions, wiki, and both platform pages.
53 changes: 53 additions & 0 deletions website/src/content/docs/platform/android.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,59 @@ grant all permissions # grants all known runtime permissions

Available permissions: `notifications`, `camera`, `location`, `microphone`, `storage`, `contacts`, `phone`, `calendar`, `sms`, `bluetooth`.

## Native UI Automation

Android is currently the only platform where probe can reach **outside the Flutter widget tree**
into native, OS-owned UI — pickers, share sheets, and any other surface the Dart agent can never
see. Elements are matched against uiautomator's text or resource-id and driven via
`uiautomator dump` + `input tap`/`input text` — no new dependencies, `adb` already ships both:

```
tap native "Choose from Gallery"
see native "IMG_0001.jpg"
don't see native "Error"
type native "wifi" into "Search settings"
```

Notes from real-device verification:

- Matching is a case-insensitive substring against both `text` and `resource-id`.
- If the native element is reached via a screen transition, add a `wait` step first — the same
idiom used after Flutter navigation.
- `take a screenshot` (the Dart-agent verb) cannot capture native UI — it renders only Flutter's
own tree. Use `adb exec-out screencap` externally to visually verify native state.
- API 35 pitfall for your own native screens: forced edge-to-edge can place a top-of-screen
element's reported bounds center underneath the app bar, which silently eats the tap — handle
window insets (`fitsSystemWindows` or equivalent).

A purpose-built native fixture app with stable ids lives in the repo under `native-test-apps/`.

## Device Media

```
add media "fixtures/photo.jpg" # adb push + MEDIA_SCANNER_SCAN_FILE broadcast
```

The file lands in `/sdcard/Pictures/` and is MediaStore-indexed, so it is genuinely visible to
image pickers — combine with the native UI verbs above to actually select it.

## Deep Links

```
open link "myapp://profile/42" in the app
```

Dispatches `am start -a android.intent.action.VIEW`, so a custom scheme or App Links URL
registered by your app is delivered to the app itself — including cold-launching it from fully
terminated (unlike iOS Simulator; see the iOS page). Plain `open link "https://..."` (no suffix)
still opens the external browser via `url_launcher`.

## Location

```
set location 48.1351, 11.5820 # adb emu geo fix (emulators only)
```

## Video Recording

Android uses the built-in `screenrecord` command. Videos are recorded as MP4 (H.264). The CLI auto-chains recordings to work around the 180-second limit.
Expand Down
43 changes: 43 additions & 0 deletions website/src/content/docs/platform/ios.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,14 @@ revoke all permissions

This works on iOS 14+ simulators.

:::note[Permission changes relaunch the app (v0.12.1+)]
`simctl privacy grant/revoke` silently **terminates the target app** — an Apple behavior, not a
probe one. As of v0.12.1, all four permission verbs automatically relaunch the app and reconnect
the session afterward, so a permission step costs a few seconds of restart instead of hanging the
next step. Follow a permission step with `wait until "<your screen>" appears` before asserting,
exactly as you would after `restart the app`.
:::

:::caution[Notification permissions cannot be pre-granted]
Apple does not support granting notification permissions via `simctl privacy`. If your app requests notification permission (e.g. via `UNUserNotificationCenter` or Firebase Messaging), the native dialog will block the Flutter UI and prevent tests from proceeding.

Expand All @@ -114,6 +122,41 @@ if (!probeEnabled) {
Build with `--dart-define=PROBE_AGENT=true` to skip these requests during testing.
:::

## Device Media

```
add media "fixtures/photo.jpg" # xcrun simctl addmedia
```

The file lands in the simulator's Photos library and is immediately visible to image pickers.
Simulators only — there is no `devicectl` equivalent for physical devices; the step skips with a
warning there.

## Deep Links

```
open link "myapp://profile/42" in the app
```

Dispatches `xcrun simctl openurl`, so a custom scheme or Universal Link registered by your app is
delivered to the app itself. Plain `open link "https://..."` still opens Safari via
`url_launcher`.

:::caution[No cold launch on iOS Simulator]
`simctl openurl` cannot launch a fully-terminated app — iOS shows an "Open in App?" confirmation
dialog that `simctl` has no way to dismiss. Deep links work reliably when the app is already
running (foreground or backgrounded); put a `restart the app` or equivalent warm-up before the
deep-link step if the app might not be running. Android has no such caveat.
:::

## Native UI Automation

The `tap native` / `see native` / `type native` verbs are **Android-only today** — iOS has no
`uiautomator` equivalent, and `simctl` cannot inspect or drive arbitrary UI. On iOS,
`tap native`/`type native` fail with a clear error rather than silently no-op'ing, and
`see native` reports "not found". iOS support via WebDriverAgent is a written, scoped proposal
(`docs/proposals/n2-ios-native-ui-bridging.md` in the repo), not yet implemented.

## Biometric Authentication (Face ID / Touch ID)

`enroll biometric`, `biometric match`, and `biometric no match` drive Face ID and Touch ID flows on iOS Simulator via `xcrun simctl spawn booted notifyutil`.
Expand Down
Loading