From 4f5a9e073e87f06bca1f2091cd82f1e2f38c8192 Mon Sep 17 00:00:00 2001 From: DongYun Kang Date: Wed, 4 Nov 2026 14:06:36 +0900 Subject: [PATCH 1/4] fix(ios): give simulator AX bridge captures a tree-bound keyboard fact The default iOS Simulator route (`simulator-ax-bridge`) published no keyboard fact. Post-gesture stabilization counts a keyboard change only between two tree-bound facts, so a drag that only dismissed the keyboard still waited for the distrust cap (about 3.7 s) on that route. The bridge reads the app's root with the same AX snapshot API as the runner, and the system keyboard window is one of the app's own windows in that tree. The adapter now publishes `observeCompleteTreeKeyboard`: the docked-tree band when the tree holds a docked keyboard, and `absent` with `treeBound` only when the tree is complete and holds no keyboard trace (no `Keyboard`/`Key` node and no iOS keyboard role, which includes the remote keyboard placeholder and the input assistant). A truncated tree, or one that holds only a trace, publishes no absence. Tests pin the adapter's fact for docked, complete, truncated, placeholder-only, and assistant-only trees; add complete-tree rows to the tap keyboard occlusion table; and drive the stabilization loop with bridge-shaped `derived -> absent` (trusts) and unchanged `absent -> absent` / `derived -> derived` pairs (fall back to content). The bridge gets its own changeset, and the `no-effect` docs and ADR 0004 now name the bridge as a follow-up to the tree-bound keyboard change. Co-Authored-By: Claude Opus 5.5 --- .changeset/ios-ax-bridge-keyboard-fact.md | 5 + .../tap-keyboard-occlusion-policy.json | 472 +++++++++++++++++- .../adr/0004-ios-snapshot-backend-strategy.md | 33 +- .../src/tap-keyboard-occlusion.test.ts | 14 + .../contracts/src/tap-keyboard-occlusion.ts | 24 + .../post-gesture-stabilization.test.ts | 83 ++- .../src/snapshot-source/adapter.test.ts | 134 ++++- .../src/snapshot-source/adapter.ts | 6 + website/docs/docs/commands.md | 2 +- 9 files changed, 766 insertions(+), 7 deletions(-) create mode 100644 .changeset/ios-ax-bridge-keyboard-fact.md diff --git a/.changeset/ios-ax-bridge-keyboard-fact.md b/.changeset/ios-ax-bridge-keyboard-fact.md new file mode 100644 index 0000000000..0b29b6a335 --- /dev/null +++ b/.changeset/ios-ax-bridge-keyboard-fact.md @@ -0,0 +1,5 @@ +--- +'agent-device': patch +--- + +Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard trace. A truncated tree stays `unmeasurable`. diff --git a/contracts/fixtures/tap-keyboard-occlusion-policy.json b/contracts/fixtures/tap-keyboard-occlusion-policy.json index 47450c03ad..a6756a3775 100644 --- a/contracts/fixtures/tap-keyboard-occlusion-policy.json +++ b/contracts/fixtures/tap-keyboard-occlusion-policy.json @@ -1,5 +1,5 @@ { - "description": "Golden table for keyboard occlusion on interaction tap points (#2589), consumed by packages/contracts/src/tap-keyboard-occlusion.ts. A visible software keyboard is its own system surface: it is never a covering sibling of app content, so the same-window occlusion classifier cannot see it, and it leaves the app window rect intact, so the viewport rule passes a covered tab bar. The band runs from the topmost keyboard node down to the bottom of the viewport, across the keyboard's own columns; the tap point is the point the interaction dispatches: the rect center the platform aims at for a ref handed over by name, or the touch point a parent with interactive descendants resolves instead. `nodes` are captured snapshot nodes as they appear in the projection the interaction resolved against \u2014 note that regular and interactive projections prune the keyboard's own container, leaving Key nodes beside app chrome. Three rules decide whether reported geometry may be measured at all. Docking: the band is derived only where the keyboard's own reported geometry reaches the viewport's bottom edge within 80 units \u2014 measured on iPhone 17 Pro (26.2) the reported key plane stops 58 pt short of an 874 pt portrait viewport and lands exactly on a 402 pt landscape one, and on iPad Pro 11-inch (M4) 3 pt short of a 1210 pt viewport \u2014 so an iPad floating keyboard, an app-drawn keypad, or an IME that stops above its own navigation bar stays unmeasured and fails open. The budget is counted in the same units as the rects it guards, which is what a measured Android IME asks for rather than what a dp table would give: with three-button navigation on Pixel 7 (Android 16, 1080x2400 at 420 dpi), Gboard's reported region runs to the physical display bottom at 2400 while the 126 px navigation bar occupies 2274..2400 inside it, so no Android keyboard measured so far needs slack for the bar. Width: geometry that comes back taller than it is wide did not arrive in the app's orientation space, i.e. it came from a capture that normalized no space \u2014 the producers that can name the app's interface orientation either turn it back (the runner's tree tiers) or refuse the capture (the Simulator AX bridge) \u2014 and is refused measurement for the same reason. Tiling: the keys themselves must form one unbroken run of columns, because a split iPad keyboard docks both clusters at the bottom edge while its surface container still spans the whole screen, and the app content between the clusters is visible and tappable. Reported seams are zero on every measured tree \u2014 iPhone 17 Pro's reported row adjoins across 395 pt, iPad Pro 11-inch's across 743.5 pt, and Gboard's across 1070 px \u2014 so the allowance covers rounding rather than layout, and a gapped band is refused measurement like the two above. That rule is deliberately blunt in both directions, and the last two cases name the trade: a sparse layout such as the iOS emoji panel reports corner keys over a wide container, does not tile, and reads `undetermined` \u2014 the guard abstains there rather than inventing which gaps are app content; and a projection that reports the container with no keys at all has nothing left to check, so the band is the container's own claim. Every captured iOS and Android keyboard measured so far reports its keys, so the second shape is the harness's, not a captured one. `target` names either a resolved node index or a bare coordinate point; both paths read one decision and choose their own consequence. A case that reaches a band declares its whole shape \u2014 the frame the refusal and the disclosure both quote, and the control rects that excuse a bare coordinate, which is where plane exclusion shows up. Provider facts distinguish measured visible frames, derived docked-tree bands, visibility-only probes, explicit absence, and unknown. Missing keyboard nodes are undetermined, never absence. A carried derived band was established before projection and stays usable after projection removes keyboard nodes; it obeys the same docking/orientation/tiling rules at acquisition. A measured floating frame needs no bottom extension.", + "description": "Golden table for keyboard occlusion on interaction tap points (#2589), consumed by packages/contracts/src/tap-keyboard-occlusion.ts. A visible software keyboard is its own system surface: it is never a covering sibling of app content, so the same-window occlusion classifier cannot see it, and it leaves the app window rect intact, so the viewport rule passes a covered tab bar. The band runs from the topmost keyboard node down to the bottom of the viewport, across the keyboard's own columns; the tap point is the point the interaction dispatches: the rect center the platform aims at for a ref handed over by name, or the touch point a parent with interactive descendants resolves instead. `nodes` are captured snapshot nodes as they appear in the projection the interaction resolved against \u2014 note that regular and interactive projections prune the keyboard's own container, leaving Key nodes beside app chrome. Three rules decide whether reported geometry may be measured at all. Docking: the band is derived only where the keyboard's own reported geometry reaches the viewport's bottom edge within 80 units \u2014 measured on iPhone 17 Pro (26.2) the reported key plane stops 58 pt short of an 874 pt portrait viewport and lands exactly on a 402 pt landscape one, and on iPad Pro 11-inch (M4) 3 pt short of a 1210 pt viewport \u2014 so an iPad floating keyboard, an app-drawn keypad, or an IME that stops above its own navigation bar stays unmeasured and fails open. The budget is counted in the same units as the rects it guards, which is what a measured Android IME asks for rather than what a dp table would give: with three-button navigation on Pixel 7 (Android 16, 1080x2400 at 420 dpi), Gboard's reported region runs to the physical display bottom at 2400 while the 126 px navigation bar occupies 2274..2400 inside it, so no Android keyboard measured so far needs slack for the bar. Width: geometry that comes back taller than it is wide did not arrive in the app's orientation space, i.e. it came from a capture that normalized no space \u2014 the producers that can name the app's interface orientation either turn it back (the runner's tree tiers) or refuse the capture (the Simulator AX bridge) \u2014 and is refused measurement for the same reason. Tiling: the keys themselves must form one unbroken run of columns, because a split iPad keyboard docks both clusters at the bottom edge while its surface container still spans the whole screen, and the app content between the clusters is visible and tappable. Reported seams are zero on every measured tree \u2014 iPhone 17 Pro's reported row adjoins across 395 pt, iPad Pro 11-inch's across 743.5 pt, and Gboard's across 1070 px \u2014 so the allowance covers rounding rather than layout, and a gapped band is refused measurement like the two above. That rule is deliberately blunt in both directions, and the last two cases name the trade: a sparse layout such as the iOS emoji panel reports corner keys over a wide container, does not tile, and reads `undetermined` \u2014 the guard abstains there rather than inventing which gaps are app content; and a projection that reports the container with no keys at all has nothing left to check, so the band is the container's own claim. Every captured iOS and Android keyboard measured so far reports its keys, so the second shape is the harness's, not a captured one. `target` names either a resolved node index or a bare coordinate point; both paths read one decision and choose their own consequence. A case that reaches a band declares its whole shape \u2014 the frame the refusal and the disclosure both quote, and the control rects that excuse a bare coordinate, which is where plane exclusion shows up. Provider facts distinguish measured visible frames, derived docked-tree bands, visibility-only probes, explicit absence, and unknown. Missing keyboard nodes are undetermined, never absence. The one exception is a capture of the app's whole in-process tree, such as the Simulator AX bridge, where the system keyboard is one of the app's own windows: `treeComplete` pins that observation, and only a complete tree with no keyboard trace at all (no `Keyboard` or `Key` node and no keyboard role, which includes the remote keyboard placeholder and the input assistant) establishes absence. A carried derived band was established before projection and stays usable after projection removes keyboard nodes; it obeys the same docking/orientation/tiling rules at acquisition. A measured floating frame needs no bottom extension.", "constants": { "occlusionReason": "tap_keyboard_occludes_target" }, @@ -5304,6 +5304,476 @@ } ] } + }, + { + "name": "a complete in-process tree with no keyboard trace establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 1 + }, + "keyboard": { + "kind": "absent", + "treeBound": true + }, + "treeComplete": true, + "expected": { + "kind": "no-keyboard" + } + }, + { + "name": "a truncated in-process tree with no keyboard nodes never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 1 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": false, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree holding only the remote keyboard placeholder never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Other", + "role": "_UIRemoteKeyboardPlaceholderView", + "identifier": "inputView", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 539, + "width": 402, + "height": 335 + } + }, + { + "index": 2, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 2 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree holding only the input assistant bar never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Other", + "role": "TUISystemInputAssistantView", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 830, + "width": 402, + "height": 44 + } + }, + { + "index": 2, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 2 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree with a docked keyboard carries its derived band, not absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "TabBar", + "label": "Tab Bar", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 791, + "width": 402, + "height": 83 + } + }, + { + "index": 2, + "type": "Button", + "label": "Form", + "parentIndex": 1, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + }, + { + "index": 3, + "type": "Key", + "label": "Q", + "parentIndex": 0, + "rect": { + "x": 4.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 4, + "type": "Key", + "label": "W", + "parentIndex": 0, + "rect": { + "x": 44, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 5, + "type": "Key", + "label": "E", + "parentIndex": 0, + "rect": { + "x": 83.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 6, + "type": "Key", + "label": "R", + "parentIndex": 0, + "rect": { + "x": 123, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 7, + "type": "Key", + "label": "T", + "parentIndex": 0, + "rect": { + "x": 162.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 8, + "type": "Key", + "label": "Y", + "parentIndex": 0, + "rect": { + "x": 202, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 9, + "type": "Key", + "label": "U", + "parentIndex": 0, + "rect": { + "x": 241.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 10, + "type": "Key", + "label": "I", + "parentIndex": 0, + "rect": { + "x": 281, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 11, + "type": "Key", + "label": "O", + "parentIndex": 0, + "rect": { + "x": 320.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 12, + "type": "Key", + "label": "P", + "parentIndex": 0, + "rect": { + "x": 360, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 13, + "type": "Key", + "label": "return", + "parentIndex": 0, + "rect": { + "x": 300, + "y": 752, + "width": 99, + "height": 54 + } + } + ], + "target": { + "index": 2 + }, + "keyboard": { + "kind": "derived", + "derivation": "docked-tree", + "frame": { + "x": 4.67, + "y": 590, + "width": 395, + "height": 284 + }, + "treeBound": true + }, + "treeComplete": true, + "expected": { + "kind": "occluded", + "frame": { + "x": 4.67, + "y": 590, + "width": 395, + "height": 284 + }, + "controlRects": [ + { + "x": 4.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 44, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 83.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 123, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 162.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 202, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 241.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 281, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 320.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 360, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 300, + "y": 752, + "width": 99, + "height": 54 + } + ] + } } ] } diff --git a/docs/adr/0004-ios-snapshot-backend-strategy.md b/docs/adr/0004-ios-snapshot-backend-strategy.md index 3362b95fd1..5c1dc711e0 100644 --- a/docs/adr/0004-ios-snapshot-backend-strategy.md +++ b/docs/adr/0004-ios-snapshot-backend-strategy.md @@ -623,9 +623,9 @@ Thus, the pass returns that tree exactly as reported. This is a structural consequence of the same rule applied uniformly. It is not the per-tier omission that a hand-threaded space would have made it. -The provider producers (`appium-source`, `limrun-ios-tree`) never see the app's windows either. -A capture from any of them publishes no keyboard fact. -A consumer reads that silence as "this producer did not measure". +The provider producers (`appium-source`, `limrun-ios-tree`, `revyl-ios-tree`) never see the app's windows either. +Their keyboard fact comes only from the docked-tree derivation or from a separate visibility probe, and it never states absence. +A consumer reads a missing or `unmeasurable` fact as "this producer did not measure". Those paths publish what the platform reported. Thus, the last reader that can still refuse geometry that it cannot place is the tap-path keyboard guard, and its width rule therefore remains. The rule detects un-normalized arrival, not a standing fact about iOS. @@ -777,3 +777,30 @@ Rejected alternatives: - Move the tree by the window origin. The bridge would then be the only correct producer for a windowed app. The keyboard window and alerts were not measured in that mode, so the refusal stays until they are. - Read the screen size on the host from the device type profile or from `simctl io`. That adds a host process and a format that Apple does not document. It also does not give the window origin. + +## Amendment: the bridge's keyboard fact (follow-up to PR #4207) + +The bridge reads the app's root with the same remote AX snapshot API as the runner. +In that tree, the system keyboard window (`UIRemoteKeyboardWindow`) is one of the app's own windows, and its frames are in app space. +The bridge sets `truncated` on every cut that it makes. + +Decision. A bridge acquisition publishes a keyboard fact (`observeCompleteTreeKeyboard`) for both intents. +A docked keyboard in the tree publishes the docked-tree band (`derived`, `treeBound`). +A complete tree (`truncated: false`) with no keyboard trace publishes `absent` with `treeBound`. +A keyboard trace is a `Keyboard` or `Key` node, or a node with an iOS keyboard role, which includes `_UIRemoteKeyboardPlaceholderView` and `TUISystemInputAssistantView`. +A truncated tree, or a complete tree that holds only a trace, publishes no absence. + +Reason. Post-gesture stabilization counts a keyboard change as fresh only between two tree-bound facts. +Without a bridge fact, a drag that only dismissed the keyboard waited for the distrust cap (about 3.5 s) on the default Simulator route. + +Rejected alternatives: + +- Publish absence from every tree without a `Keyboard` or `Key` node. A minimized keyboard, a placeholder that outlives the key plane during a dismissal, and an out-of-process keyboard can leave only a trace. Absence there could end the distrust window on a stale read. +- Publish absence from a truncated tree. A cut can drop the keyboard window. + +Consequences. `snapshot --json` shows `keyboard` for bridge captures. +On a complete tree with no keyboard trace, the tap guard reads `no-keyboard` instead of `undetermined`. +Only `occluded` changes a tap, so taps do not change. +The bridge band excludes the input assistant bar, as the band that the tap guard derived for bridge captures already did. +The runner's measured band includes that bar. +Open gaps: third-party keyboards, iPad floating and split keyboards, and hardware-keyboard states were not captured on the bridge. diff --git a/packages/contracts/src/tap-keyboard-occlusion.test.ts b/packages/contracts/src/tap-keyboard-occlusion.test.ts index ce5c07b4dd..22af08f474 100644 --- a/packages/contracts/src/tap-keyboard-occlusion.test.ts +++ b/packages/contracts/src/tap-keyboard-occlusion.test.ts @@ -9,6 +9,7 @@ import type { } from '@agent-device/kernel/snapshot'; import { acquireProviderKeyboard, + observeCompleteTreeKeyboard, resolveKeyboardTapOcclusion, TAP_KEYBOARD_OCCLUDES_TARGET_DETAILS, TAP_KEYBOARD_OCCLUDES_TARGET_REASON, @@ -34,6 +35,8 @@ type FixtureCase = { target: { index: number } | { point: { x: number; y: number } }; keyboard?: SnapshotKeyboardBandFact; observationVisibility?: boolean; + /** Whether a producer that reads the app's whole in-process tree captured `nodes` uncut. */ + treeComplete?: boolean; expected: { kind: string; frame?: Rect; controlRects?: readonly Rect[] }; }; @@ -84,6 +87,17 @@ test('keyboard tap occlusion agrees with every golden parity table case', () => `${fixture.name}: acquisition observation`, ); } + if (fixture.treeComplete !== undefined) { + assert.deepEqual( + observeCompleteTreeKeyboard({ + nodes, + viewport: fixture.viewport, + complete: fixture.treeComplete, + }), + fixture.keyboard, + `${fixture.name}: complete-tree observation`, + ); + } const occlusion = resolveKeyboardTapOcclusion({ nodes, viewport: fixture.viewport, diff --git a/packages/contracts/src/tap-keyboard-occlusion.ts b/packages/contracts/src/tap-keyboard-occlusion.ts index f332649468..5e0ccf6e19 100644 --- a/packages/contracts/src/tap-keyboard-occlusion.ts +++ b/packages/contracts/src/tap-keyboard-occlusion.ts @@ -356,6 +356,30 @@ export function acquireProviderKeyboard(params: ProviderKeyboardInput): { }; } +/** + * The keyboard fact of a producer that reads the target app's whole in-process tree in one capture, + * as the Simulator AX bridge does: the keyboard window is one of the app's own windows there, so a + * complete tree with no keyboard node proves absence the way a full XCTest scan does. A truncated tree + * proves only presence. + */ +export function observeCompleteTreeKeyboard( + params: Readonly<{ nodes: readonly RawSnapshotNode[]; viewport: Rect | null; complete: boolean }>, +): SnapshotKeyboardBandFact { + const fact = observeProviderKeyboard(params); + if (fact.kind !== 'unmeasurable' || !params.complete) return fact; + return params.nodes.some(isKeyboardTraceNode) ? fact : { kind: 'absent', treeBound: true }; +} + +/** Any node the keyboard leaves in the tree, including the in-app placeholder of a remote keyboard. */ +function isKeyboardTraceNode(node: RawSnapshotNode): boolean { + return ( + isKeyboardSurfaceNode(node) || + hasIosKeyboardRole(node, IOS_KEYBOARD_SETTLE_CHROME_ROLES) || + hasIosKeyboardRole(node, IOS_KEYBOARD_SUBTREE_CHROME_ROLES) || + hasIosKeyboardRole(node, IOS_KEYBOARD_CHROME_MARKER_ROLES) + ); +} + function observeProviderKeyboard(params: ProviderKeyboardInput): SnapshotKeyboardBandFact { const frame = deriveDockedKeyboardFrame(params.nodes, params.viewport); // Derived from the nodes this capture returns, so a stale tree derives the stale tree's band. diff --git a/packages/daemon/src/core/__tests__/post-gesture-stabilization.test.ts b/packages/daemon/src/core/__tests__/post-gesture-stabilization.test.ts index 3d774a15a1..727bdfec75 100644 --- a/packages/daemon/src/core/__tests__/post-gesture-stabilization.test.ts +++ b/packages/daemon/src/core/__tests__/post-gesture-stabilization.test.ts @@ -1,5 +1,8 @@ import type { CommandFlags } from '@agent-device/contracts/command'; -import { acquireProviderKeyboard } from '@agent-device/contracts/tap-keyboard-occlusion'; +import { + acquireProviderKeyboard, + observeCompleteTreeKeyboard, +} from '@agent-device/contracts/tap-keyboard-occlusion'; import assert from 'node:assert/strict'; import { afterEach, test, vi } from 'vitest'; import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures'; @@ -608,6 +611,84 @@ test('capturePostGestureStabilizedResult still distrusts a stale WebDriver sourc assert.equal(settled, 0); }); +/** A Simulator AX bridge capture: one complete read of the app's tree, keyboard window included. */ +function bridgeCapture(keyboard: boolean) { + const source = makeSnapshotState([ + ...pickupSnapshot(500).nodes, + ...(keyboard ? keyboardWindowNodes() : []), + ]); + return { + ...source, + keyboard: observeCompleteTreeKeyboard({ + nodes: source.nodes, + viewport: { x: 0, y: 0, width: 390, height: 844 }, + complete: true, + }), + }; +} + +async function stabilizeBridgeCaptures(params: { before: boolean; after: boolean }) { + vi.useFakeTimers(); + const session = makeSession('ios'); + session.snapshot = bridgeCapture(params.before); + markPostGestureStabilization(session, 'swipe'); + const baselineKeyboard = session.snapshot.keyboard; + const capture = vi.fn(async () => bridgeCapture(params.after)); + + const resultPromise = withDiagnosticsScope({}, async () => { + const result = await capturePostGestureStabilizedResult({ + session, + capture, + readSnapshot: (snapshot) => snapshot, + }); + return { + result, + staleAccepts: countDiagnosticEventsByPhase(['post_gesture_snapshot_stale_accept']), + settled: countDiagnosticEventsByPhase(['post_gesture_snapshot_stabilized']), + }; + }); + + await vi.advanceTimersByTimeAsync(10_000); + const { result, staleAccepts, settled } = await resultPromise; + return { + staleAccepts, + settled, + baselineKeyboard, + quietKeyboard: result.value.keyboard, + postGestureOutcome: result.postGestureOutcome, + captures: capture.mock.calls.length, + }; +} + +test('capturePostGestureStabilizedResult settles when a complete bridge tree loses the keyboard it derived (iOS)', async () => { + const outcome = await stabilizeBridgeCaptures({ before: true, after: false }); + + assert.deepEqual(outcome.baselineKeyboard, { + kind: 'derived', + derivation: 'docked-tree', + frame: { x: 0, y: 500, width: 390, height: 344 }, + treeBound: true, + }); + assert.deepEqual(outcome.quietKeyboard, { kind: 'absent', treeBound: true }); + assert.equal(outcome.settled, 1); + assert.equal(outcome.staleAccepts, 0); + assert.equal(outcome.postGestureOutcome, undefined); + assert.equal(outcome.captures, 2); +}); + +test('capturePostGestureStabilizedResult still distrusts bridge trees whose keyboard fact did not change (iOS)', async () => { + // The same fact on both sides proves nothing, so the content comparison decides, and it never moved. + for (const keyboard of [false, true]) { + const outcome = await stabilizeBridgeCaptures({ before: keyboard, after: keyboard }); + + assert.equal(outcome.quietKeyboard?.kind, keyboard ? 'derived' : 'absent'); + assert.deepEqual(outcome.quietKeyboard, outcome.baselineKeyboard); + assert.equal(outcome.staleAccepts, 1, `keyboard ${keyboard}`); + assert.equal(outcome.settled, 0, `keyboard ${keyboard}`); + vi.useRealTimers(); + } +}); + // --- #1569: a backend swap mid-poll is not comparable evidence --- test('capturePostGestureStabilizedResult re-baselines instead of concluding when the backend changes (iOS)', async () => { diff --git a/packages/platform-apple/src/snapshot-source/adapter.test.ts b/packages/platform-apple/src/snapshot-source/adapter.test.ts index 7f4ecfa446..ca8bf21426 100644 --- a/packages/platform-apple/src/snapshot-source/adapter.test.ts +++ b/packages/platform-apple/src/snapshot-source/adapter.test.ts @@ -554,6 +554,8 @@ type AdapterFixture = { runningSurfaceHost: boolean; /** The surface host check each request carried: its executables and Simulator, or undefined. */ surfaceHostChecks: Array | undefined>; + /** The windows the fake guest reports under its root when no other tree shape is selected. */ + windows: readonly Record[]; omitRecovery: boolean; diagnostics: Record[]; startedTargets: Array[0]>; @@ -591,6 +593,7 @@ function createAdapterHost(buildDelayMs = 0): AdapterFixture { omitViewport: false, runningSurfaceHost: false, surfaceHostChecks: [], + windows: [], omitRecovery: false, diagnostics: [], startedTargets: [], @@ -813,7 +816,7 @@ class AdapterSocket extends EventEmitter implements SnapshotSourceSocket { ], }, ] - : [], + : this.fixture.windows, }, }, { @@ -873,3 +876,132 @@ test('the Simulator AX source refuses a window whose coordinate space it cannot await rm(root, { recursive: true, force: true }); } }); + +/** A guest node; `automationType` 19 is `Keyboard` and 20 is `Key`. */ +function guestNode( + elementType: string, + frame: Readonly<{ X: number; Y: number; Width: number; Height: number }>, + children: readonly Record[] = [], + automationType?: number, +): Record { + return { + XC_kAXXCAttributeElementType: elementType, + ...(automationType === undefined ? {} : { XC_kAXXCAttributeAutomationType: automationType }), + XC_kAXXCAttributeFrame: frame, + XC_kAXXCAttributeChildren: children, + }; +} + +const APP_FRAME = { X: 0, Y: 0, Width: 390, Height: 844 }; +const ASSISTANT_FRAME = { X: 0, Y: 509, Width: 390, Height: 44 }; +// The docked key plane stops 58 pt above the viewport bottom, as on iPhone 17 Pro. +const DOCKED_KEYBOARD_WINDOW = guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ + guestNode('TUISystemInputAssistantView', ASSISTANT_FRAME), + guestNode( + 'UIKeyboardAutomatic', + { X: 0, Y: 553, Width: 390, Height: 233 }, + [ + guestNode('UIKeyboardLayoutStar', { X: 0, Y: 553, Width: 390, Height: 233 }, [ + guestNode('UIAccessibilityElementKBKey', { X: 0, Y: 560, Width: 195, Height: 54 }, [], 20), + guestNode( + 'UIAccessibilityElementKBKey', + { X: 195, Y: 560, Width: 195, Height: 54 }, + [], + 20, + ), + ]), + ], + 19, + ), +]); + +test('the Simulator AX source publishes absence only from a complete tree with no keyboard trace', async () => { + const root = await mkdtempForTest('agent-device-snapshot-adapter-keyboard-'); + const sourceRoot = path.join(root, 'source'); + await (await import('@agent-device/host-kit/host-file')).ensureHostDirectory(sourceRoot); + for (const name of [ + 'SnapshotBridge.m', + 'SnapshotBridgeRuntime.m', + 'SnapshotBridgeRuntime.h', + 'SnapshotBridgeCapture.h', + 'SnapshotBridgeCapture.m', + 'ForegroundApplicationGeometry.h', + ]) { + await writeFile(path.join(sourceRoot, name), 'native source'); + } + const fixture = createAdapterHost(); + const source = createSimulatorSnapshotSource({ + host: fixture.host, + sourceRoot, + cacheRoot: path.join(root, 'cache'), + }); + const target = { ...targetForTest(), generation: 'generation-1', targetId: 'target-1' }; + const notObserved = { kind: 'unmeasurable', reason: 'keyboard-not-observed' }; + const cases = [ + { + name: 'a docked keyboard', + windows: [DOCKED_KEYBOARD_WINDOW], + truncated: false, + keyboard: { + kind: 'derived', + derivation: 'docked-tree', + frame: { x: 0, y: 553, width: 390, height: 291 }, + treeBound: true, + }, + }, + { + name: 'a complete tree', + windows: [], + truncated: false, + keyboard: { kind: 'absent', treeBound: true }, + }, + // A cut can drop the keyboard window, so a truncated tree proves only presence. + { name: 'a truncated tree', windows: [], truncated: true, keyboard: notObserved }, + // The in-app placeholder can outlive the key plane during a dismissal. + { + name: 'a placeholder without keys', + windows: [ + guestNode('UITextEffectsWindow', APP_FRAME, [ + guestNode('_UIRemoteKeyboardPlaceholderView', { X: 0, Y: 509, Width: 390, Height: 335 }), + ]), + ], + truncated: false, + keyboard: notObserved, + }, + // A minimized keyboard leaves only its assistant bar. + { + name: 'an assistant bar without keys', + windows: [ + guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ + guestNode('TUISystemInputAssistantView', ASSISTANT_FRAME), + ]), + ], + truncated: false, + keyboard: notObserved, + }, + ]; + + try { + for (const entry of cases) { + fixture.windows = entry.windows; + fixture.truncated = entry.truncated; + for (const acquisitionIntent of ['full', 'surface-observation'] as const) { + const outcome = await source.acquire({ + target, + hint: deriveIosCaptureHint(createIosSnapshotRequest({ acquisitionIntent })), + }); + assert.equal(outcome.stage, 'acquired', `${entry.name} (${acquisitionIntent})`); + if (outcome.stage !== 'acquired') continue; + assert.equal(outcome.acquisition.truncated, entry.truncated); + assert.deepEqual( + outcome.acquisition.keyboard, + entry.keyboard, + `${entry.name} (${acquisitionIntent})`, + ); + } + } + } finally { + await source.close(); + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/packages/platform-apple/src/snapshot-source/adapter.ts b/packages/platform-apple/src/snapshot-source/adapter.ts index a62ac5ea1e..9540a7acf3 100644 --- a/packages/platform-apple/src/snapshot-source/adapter.ts +++ b/packages/platform-apple/src/snapshot-source/adapter.ts @@ -1,4 +1,5 @@ import { AppError } from '@agent-device/kernel/errors'; +import { observeCompleteTreeKeyboard } from '@agent-device/contracts/tap-keyboard-occlusion'; import type { CaptureHint, IosSnapshotAcquisition, @@ -246,6 +247,11 @@ function createAcquisition( nodes, truncated, viewport: decoded.viewport, + keyboard: observeCompleteTreeKeyboard({ + nodes, + viewport: decoded.viewport.kind === 'reported' ? decoded.viewport.rect : null, + complete: !truncated, + }), lineage, residue, }; diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index ecce60609b..d0f577e2f7 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -607,7 +607,7 @@ agent-device is text 'id="greeting"' "Welcome back" - On the web, on-screen text the browser cannot hand an action to (static text, a `status` announcement) is in the tree and readable by `wait text`, `find text`, `get text`, and `text=`, but carries no `@ref` and has no rect: `find ... list` prints it as `(no ref)`, and it is never a tap target. Such text is published on `value`, so `value="X"` can match page text as well as a field holding that text, and a predicate that needs one element then reports `AMBIGUOUS_MATCH`; use `text=` for page text, or add `editable=true` to name the field. - The first read after `scroll`, `swipe`, or `gesture swipe` waits for two captures to agree. When that proves something about the gesture, `is`, `get`, `find`, `wait`, interactions that captured it, and `snapshot` add a warning, and all but `snapshot` report `postGestureOutcome` (`{ "kind", "gesture": { "action", "positionals" } }`) in `data` or `error.details`: - `kind: "unsettled"`: the screen was still changing. A miss is not proof of absence; read again. `is absent` refuses with `observation: "unsettled"`, and `wait absent` keeps polling. - - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner does, a WebDriver (Appium) visibility probe does not. Otherwise only the content is compared. + - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree. Otherwise only the content is compared. ## Replay From 0c987a6fc3d8037bdfe199c45de3a32919402c36 Mon Sep 17 00:00:00 2001 From: DongYun Kang Date: Wed, 4 Nov 2026 14:34:33 +0900 Subject: [PATCH 2/4] fix(ios): keep a keyboard window without keys from proving absence A complete Simulator AX bridge tree that held a `UIRemoteKeyboardWindow` with no keys published `absent` with `treeBound`: the window class was not a keyboard trace. A keyboard window that is present but exposes no keys, for example a hardware keyboard that shows only the assistant bar, is a partial reading, so it must stay `unmeasurable`. The keyboard window now counts as a keyboard trace. Only a complete tree with no keyboard window, no `Keyboard` or `Key` node, and no keyboard role publishes absence. Adapter tests and golden rows pin a keyboard window without keys, with only the assistant bar, and with only the remote keyboard placeholder. The changeset, the `no-effect` docs, and ADR 0004 name the keyboard window. Co-Authored-By: Claude Opus 5.5 --- .changeset/ios-ax-bridge-keyboard-fact.md | 2 +- .../tap-keyboard-occlusion-policy.json | 194 +++++++++++++++++- .../adr/0004-ios-snapshot-backend-strategy.md | 5 +- .../contracts/src/tap-keyboard-occlusion.ts | 9 +- .../src/snapshot-source/adapter.test.ts | 19 +- website/docs/docs/commands.md | 2 +- 6 files changed, 222 insertions(+), 9 deletions(-) diff --git a/.changeset/ios-ax-bridge-keyboard-fact.md b/.changeset/ios-ax-bridge-keyboard-fact.md index 0b29b6a335..2521f5d228 100644 --- a/.changeset/ios-ax-bridge-keyboard-fact.md +++ b/.changeset/ios-ax-bridge-keyboard-fact.md @@ -2,4 +2,4 @@ 'agent-device': patch --- -Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard trace. A truncated tree stays `unmeasurable`. +Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree, or a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar), stays `unmeasurable`. diff --git a/contracts/fixtures/tap-keyboard-occlusion-policy.json b/contracts/fixtures/tap-keyboard-occlusion-policy.json index a6756a3775..9b63b502a9 100644 --- a/contracts/fixtures/tap-keyboard-occlusion-policy.json +++ b/contracts/fixtures/tap-keyboard-occlusion-policy.json @@ -1,5 +1,5 @@ { - "description": "Golden table for keyboard occlusion on interaction tap points (#2589), consumed by packages/contracts/src/tap-keyboard-occlusion.ts. A visible software keyboard is its own system surface: it is never a covering sibling of app content, so the same-window occlusion classifier cannot see it, and it leaves the app window rect intact, so the viewport rule passes a covered tab bar. The band runs from the topmost keyboard node down to the bottom of the viewport, across the keyboard's own columns; the tap point is the point the interaction dispatches: the rect center the platform aims at for a ref handed over by name, or the touch point a parent with interactive descendants resolves instead. `nodes` are captured snapshot nodes as they appear in the projection the interaction resolved against \u2014 note that regular and interactive projections prune the keyboard's own container, leaving Key nodes beside app chrome. Three rules decide whether reported geometry may be measured at all. Docking: the band is derived only where the keyboard's own reported geometry reaches the viewport's bottom edge within 80 units \u2014 measured on iPhone 17 Pro (26.2) the reported key plane stops 58 pt short of an 874 pt portrait viewport and lands exactly on a 402 pt landscape one, and on iPad Pro 11-inch (M4) 3 pt short of a 1210 pt viewport \u2014 so an iPad floating keyboard, an app-drawn keypad, or an IME that stops above its own navigation bar stays unmeasured and fails open. The budget is counted in the same units as the rects it guards, which is what a measured Android IME asks for rather than what a dp table would give: with three-button navigation on Pixel 7 (Android 16, 1080x2400 at 420 dpi), Gboard's reported region runs to the physical display bottom at 2400 while the 126 px navigation bar occupies 2274..2400 inside it, so no Android keyboard measured so far needs slack for the bar. Width: geometry that comes back taller than it is wide did not arrive in the app's orientation space, i.e. it came from a capture that normalized no space \u2014 the producers that can name the app's interface orientation either turn it back (the runner's tree tiers) or refuse the capture (the Simulator AX bridge) \u2014 and is refused measurement for the same reason. Tiling: the keys themselves must form one unbroken run of columns, because a split iPad keyboard docks both clusters at the bottom edge while its surface container still spans the whole screen, and the app content between the clusters is visible and tappable. Reported seams are zero on every measured tree \u2014 iPhone 17 Pro's reported row adjoins across 395 pt, iPad Pro 11-inch's across 743.5 pt, and Gboard's across 1070 px \u2014 so the allowance covers rounding rather than layout, and a gapped band is refused measurement like the two above. That rule is deliberately blunt in both directions, and the last two cases name the trade: a sparse layout such as the iOS emoji panel reports corner keys over a wide container, does not tile, and reads `undetermined` \u2014 the guard abstains there rather than inventing which gaps are app content; and a projection that reports the container with no keys at all has nothing left to check, so the band is the container's own claim. Every captured iOS and Android keyboard measured so far reports its keys, so the second shape is the harness's, not a captured one. `target` names either a resolved node index or a bare coordinate point; both paths read one decision and choose their own consequence. A case that reaches a band declares its whole shape \u2014 the frame the refusal and the disclosure both quote, and the control rects that excuse a bare coordinate, which is where plane exclusion shows up. Provider facts distinguish measured visible frames, derived docked-tree bands, visibility-only probes, explicit absence, and unknown. Missing keyboard nodes are undetermined, never absence. The one exception is a capture of the app's whole in-process tree, such as the Simulator AX bridge, where the system keyboard is one of the app's own windows: `treeComplete` pins that observation, and only a complete tree with no keyboard trace at all (no `Keyboard` or `Key` node and no keyboard role, which includes the remote keyboard placeholder and the input assistant) establishes absence. A carried derived band was established before projection and stays usable after projection removes keyboard nodes; it obeys the same docking/orientation/tiling rules at acquisition. A measured floating frame needs no bottom extension.", + "description": "Golden table for keyboard occlusion on interaction tap points (#2589), consumed by packages/contracts/src/tap-keyboard-occlusion.ts. A visible software keyboard is its own system surface: it is never a covering sibling of app content, so the same-window occlusion classifier cannot see it, and it leaves the app window rect intact, so the viewport rule passes a covered tab bar. The band runs from the topmost keyboard node down to the bottom of the viewport, across the keyboard's own columns; the tap point is the point the interaction dispatches: the rect center the platform aims at for a ref handed over by name, or the touch point a parent with interactive descendants resolves instead. `nodes` are captured snapshot nodes as they appear in the projection the interaction resolved against \u2014 note that regular and interactive projections prune the keyboard's own container, leaving Key nodes beside app chrome. Three rules decide whether reported geometry may be measured at all. Docking: the band is derived only where the keyboard's own reported geometry reaches the viewport's bottom edge within 80 units \u2014 measured on iPhone 17 Pro (26.2) the reported key plane stops 58 pt short of an 874 pt portrait viewport and lands exactly on a 402 pt landscape one, and on iPad Pro 11-inch (M4) 3 pt short of a 1210 pt viewport \u2014 so an iPad floating keyboard, an app-drawn keypad, or an IME that stops above its own navigation bar stays unmeasured and fails open. The budget is counted in the same units as the rects it guards, which is what a measured Android IME asks for rather than what a dp table would give: with three-button navigation on Pixel 7 (Android 16, 1080x2400 at 420 dpi), Gboard's reported region runs to the physical display bottom at 2400 while the 126 px navigation bar occupies 2274..2400 inside it, so no Android keyboard measured so far needs slack for the bar. Width: geometry that comes back taller than it is wide did not arrive in the app's orientation space, i.e. it came from a capture that normalized no space \u2014 the producers that can name the app's interface orientation either turn it back (the runner's tree tiers) or refuse the capture (the Simulator AX bridge) \u2014 and is refused measurement for the same reason. Tiling: the keys themselves must form one unbroken run of columns, because a split iPad keyboard docks both clusters at the bottom edge while its surface container still spans the whole screen, and the app content between the clusters is visible and tappable. Reported seams are zero on every measured tree \u2014 iPhone 17 Pro's reported row adjoins across 395 pt, iPad Pro 11-inch's across 743.5 pt, and Gboard's across 1070 px \u2014 so the allowance covers rounding rather than layout, and a gapped band is refused measurement like the two above. That rule is deliberately blunt in both directions, and the last two cases name the trade: a sparse layout such as the iOS emoji panel reports corner keys over a wide container, does not tile, and reads `undetermined` \u2014 the guard abstains there rather than inventing which gaps are app content; and a projection that reports the container with no keys at all has nothing left to check, so the band is the container's own claim. Every captured iOS and Android keyboard measured so far reports its keys, so the second shape is the harness's, not a captured one. `target` names either a resolved node index or a bare coordinate point; both paths read one decision and choose their own consequence. A case that reaches a band declares its whole shape \u2014 the frame the refusal and the disclosure both quote, and the control rects that excuse a bare coordinate, which is where plane exclusion shows up. Provider facts distinguish measured visible frames, derived docked-tree bands, visibility-only probes, explicit absence, and unknown. Missing keyboard nodes are undetermined, never absence. The one exception is a capture of the app's whole in-process tree, such as the Simulator AX bridge, where the system keyboard is one of the app's own windows: `treeComplete` pins that observation, and only a complete tree with no keyboard trace at all (no keyboard window, no `Keyboard` or `Key` node, and no keyboard role, which includes the remote keyboard placeholder and the input assistant) establishes absence; a keyboard window that exposes no keys stays undetermined. A carried derived band was established before projection and stays usable after projection removes keyboard nodes; it obeys the same docking/orientation/tiling rules at acquisition. A measured floating frame needs no bottom extension.", "constants": { "occlusionReason": "tap_keyboard_occludes_target" }, @@ -5506,6 +5506,198 @@ "kind": "undetermined" } }, + { + "name": "a complete in-process tree holding a keyboard window without keys never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Window", + "role": "UIRemoteKeyboardWindow", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 2, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 2 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree holding a keyboard window with only the input assistant bar never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Window", + "role": "UIRemoteKeyboardWindow", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 2, + "type": "Other", + "role": "TUISystemInputAssistantView", + "parentIndex": 1, + "rect": { + "x": 0, + "y": 830, + "width": 402, + "height": 44 + } + }, + { + "index": 3, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 3 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree holding a keyboard window with only the remote keyboard placeholder never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Window", + "role": "UIRemoteKeyboardWindow", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 2, + "type": "Other", + "role": "_UIRemoteKeyboardPlaceholderView", + "parentIndex": 1, + "rect": { + "x": 0, + "y": 539, + "width": 402, + "height": 335 + } + }, + { + "index": 3, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 3 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, { "name": "a complete in-process tree with a docked keyboard carries its derived band, not absence", "viewport": { diff --git a/docs/adr/0004-ios-snapshot-backend-strategy.md b/docs/adr/0004-ios-snapshot-backend-strategy.md index 5c1dc711e0..ae0d995ab5 100644 --- a/docs/adr/0004-ios-snapshot-backend-strategy.md +++ b/docs/adr/0004-ios-snapshot-backend-strategy.md @@ -787,8 +787,9 @@ The bridge sets `truncated` on every cut that it makes. Decision. A bridge acquisition publishes a keyboard fact (`observeCompleteTreeKeyboard`) for both intents. A docked keyboard in the tree publishes the docked-tree band (`derived`, `treeBound`). A complete tree (`truncated: false`) with no keyboard trace publishes `absent` with `treeBound`. -A keyboard trace is a `Keyboard` or `Key` node, or a node with an iOS keyboard role, which includes `_UIRemoteKeyboardPlaceholderView` and `TUISystemInputAssistantView`. -A truncated tree, or a complete tree that holds only a trace, publishes no absence. +A keyboard trace is the keyboard window (`UIRemoteKeyboardWindow`), a `Keyboard` or `Key` node, or a node with an iOS keyboard role, which includes `_UIRemoteKeyboardPlaceholderView` and `TUISystemInputAssistantView`. +A truncated tree, or a complete tree that holds only a trace, publishes `unmeasurable`. +Thus, a keyboard window that exposes no keys, for example a hardware keyboard that shows only the assistant bar, never publishes absence. Reason. Post-gesture stabilization counts a keyboard change as fresh only between two tree-bound facts. Without a bridge fact, a drag that only dismissed the keyboard waited for the distrust cap (about 3.5 s) on the default Simulator route. diff --git a/packages/contracts/src/tap-keyboard-occlusion.ts b/packages/contracts/src/tap-keyboard-occlusion.ts index 5e0ccf6e19..3202e01823 100644 --- a/packages/contracts/src/tap-keyboard-occlusion.ts +++ b/packages/contracts/src/tap-keyboard-occlusion.ts @@ -66,6 +66,8 @@ export const TAP_KEYBOARD_OCCLUDES_TARGET_DETAILS = Object.freeze({ }); const IOS_KEYBOARD_TYPE_NAMES: ReadonlySet = new Set(['keyboard', 'key']); +/** The system window that hosts the remote keyboard, with or without keys in it. */ +const IOS_KEYBOARD_WINDOW_ROLE = 'UIRemoteKeyboardWindow'; /** The occluding band of the visible keyboard, plus the rects that prove a tap meant the keyboard. */ export type KeyboardSurface = { @@ -359,8 +361,8 @@ export function acquireProviderKeyboard(params: ProviderKeyboardInput): { /** * The keyboard fact of a producer that reads the target app's whole in-process tree in one capture, * as the Simulator AX bridge does: the keyboard window is one of the app's own windows there, so a - * complete tree with no keyboard node proves absence the way a full XCTest scan does. A truncated tree - * proves only presence. + * complete tree with no keyboard window and no keyboard node proves absence the way a full XCTest scan + * does. A truncated tree, or a keyboard window that exposes no keys, proves only presence. */ export function observeCompleteTreeKeyboard( params: Readonly<{ nodes: readonly RawSnapshotNode[]; viewport: Rect | null; complete: boolean }>, @@ -370,9 +372,10 @@ export function observeCompleteTreeKeyboard( return params.nodes.some(isKeyboardTraceNode) ? fact : { kind: 'absent', treeBound: true }; } -/** Any node the keyboard leaves in the tree, including the in-app placeholder of a remote keyboard. */ +/** Any node the keyboard leaves in the tree, including its window and the in-app placeholder. */ function isKeyboardTraceNode(node: RawSnapshotNode): boolean { return ( + node.role === IOS_KEYBOARD_WINDOW_ROLE || isKeyboardSurfaceNode(node) || hasIosKeyboardRole(node, IOS_KEYBOARD_SETTLE_CHROME_ROLES) || hasIosKeyboardRole(node, IOS_KEYBOARD_SUBTREE_CHROME_ROLES) || diff --git a/packages/platform-apple/src/snapshot-source/adapter.test.ts b/packages/platform-apple/src/snapshot-source/adapter.test.ts index ca8bf21426..5921796a07 100644 --- a/packages/platform-apple/src/snapshot-source/adapter.test.ts +++ b/packages/platform-apple/src/snapshot-source/adapter.test.ts @@ -968,7 +968,24 @@ test('the Simulator AX source publishes absence only from a complete tree with n truncated: false, keyboard: notObserved, }, - // A minimized keyboard leaves only its assistant bar. + // A keyboard window that exposes no keys is not proof that the keyboard is gone. + { + name: 'a keyboard window without keys', + windows: [guestNode('UIRemoteKeyboardWindow', APP_FRAME)], + truncated: false, + keyboard: notObserved, + }, + { + name: 'a keyboard window that holds only a placeholder', + windows: [ + guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ + guestNode('_UIRemoteKeyboardPlaceholderView', { X: 0, Y: 509, Width: 390, Height: 335 }), + ]), + ], + truncated: false, + keyboard: notObserved, + }, + // A minimized or hardware keyboard leaves only its assistant bar. { name: 'an assistant bar without keys', windows: [ diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index d0f577e2f7..29b9116c00 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -607,7 +607,7 @@ agent-device is text 'id="greeting"' "Welcome back" - On the web, on-screen text the browser cannot hand an action to (static text, a `status` announcement) is in the tree and readable by `wait text`, `find text`, `get text`, and `text=`, but carries no `@ref` and has no rect: `find ... list` prints it as `(no ref)`, and it is never a tap target. Such text is published on `value`, so `value="X"` can match page text as well as a field holding that text, and a predicate that needs one element then reports `AMBIGUOUS_MATCH`; use `text=` for page text, or add `editable=true` to name the field. - The first read after `scroll`, `swipe`, or `gesture swipe` waits for two captures to agree. When that proves something about the gesture, `is`, `get`, `find`, `wait`, interactions that captured it, and `snapshot` add a warning, and all but `snapshot` report `postGestureOutcome` (`{ "kind", "gesture": { "action", "positionals" } }`) in `data` or `error.details`: - `kind: "unsettled"`: the screen was still changing. A miss is not proof of absence; read again. `is absent` refuses with `observation: "unsettled"`, and `wait absent` keeps polling. - - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree. Otherwise only the content is compared. + - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window. Otherwise only the content is compared. ## Replay From 4d0176d5f1e98183a24c35a523f7c0d7ce1056f3 Mon Sep 17 00:00:00 2001 From: DongYun Kang Date: Wed, 4 Nov 2026 21:52:55 +0900 Subject: [PATCH 3/4] test(ios): move the AX bridge keyboard cases into a fixtures file The rebase onto the surface-host check took adapter.test.ts past the 1000-line cap. The guest trees now live beside it, and the setup reads SNAPSHOT_BRIDGE_SOURCE_FILENAMES, so it also writes the surface-host sources. Co-Authored-By: Claude Opus 5.5 --- .../adapter-keyboard.fixtures.ts | 110 +++++++++++++++++ .../src/snapshot-source/adapter.test.ts | 111 +----------------- 2 files changed, 113 insertions(+), 108 deletions(-) create mode 100644 packages/platform-apple/src/snapshot-source/adapter-keyboard.fixtures.ts diff --git a/packages/platform-apple/src/snapshot-source/adapter-keyboard.fixtures.ts b/packages/platform-apple/src/snapshot-source/adapter-keyboard.fixtures.ts new file mode 100644 index 0000000000..2ed559dbf0 --- /dev/null +++ b/packages/platform-apple/src/snapshot-source/adapter-keyboard.fixtures.ts @@ -0,0 +1,110 @@ +/** + * Guest trees for the Simulator AX source's keyboard fact: each case names the windows the fake + * guest reports under its root, whether it cut the tree, and the fact the capture must publish. + */ + +/** A guest node; `automationType` 19 is `Keyboard` and 20 is `Key`. */ +function guestNode( + elementType: string, + frame: Readonly<{ X: number; Y: number; Width: number; Height: number }>, + children: readonly Record[] = [], + automationType?: number, +): Record { + return { + XC_kAXXCAttributeElementType: elementType, + ...(automationType === undefined ? {} : { XC_kAXXCAttributeAutomationType: automationType }), + XC_kAXXCAttributeFrame: frame, + XC_kAXXCAttributeChildren: children, + }; +} + +const APP_FRAME = { X: 0, Y: 0, Width: 390, Height: 844 }; +const ASSISTANT_FRAME = { X: 0, Y: 509, Width: 390, Height: 44 }; +// The docked key plane stops 58 pt above the viewport bottom, as on iPhone 17 Pro. +const DOCKED_KEYBOARD_WINDOW = guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ + guestNode('TUISystemInputAssistantView', ASSISTANT_FRAME), + guestNode( + 'UIKeyboardAutomatic', + { X: 0, Y: 553, Width: 390, Height: 233 }, + [ + guestNode('UIKeyboardLayoutStar', { X: 0, Y: 553, Width: 390, Height: 233 }, [ + guestNode('UIAccessibilityElementKBKey', { X: 0, Y: 560, Width: 195, Height: 54 }, [], 20), + guestNode( + 'UIAccessibilityElementKBKey', + { X: 195, Y: 560, Width: 195, Height: 54 }, + [], + 20, + ), + ]), + ], + 19, + ), +]); + +const notObserved = { kind: 'unmeasurable', reason: 'keyboard-not-observed' }; + +export const ADAPTER_KEYBOARD_CASES: readonly Readonly<{ + name: string; + windows: readonly Record[]; + truncated: boolean; + keyboard: Record; +}>[] = [ + { + name: 'a docked keyboard', + windows: [DOCKED_KEYBOARD_WINDOW], + truncated: false, + keyboard: { + kind: 'derived', + derivation: 'docked-tree', + frame: { x: 0, y: 553, width: 390, height: 291 }, + treeBound: true, + }, + }, + { + name: 'a complete tree', + windows: [], + truncated: false, + keyboard: { kind: 'absent', treeBound: true }, + }, + // A cut can drop the keyboard window, so a truncated tree proves only presence. + { name: 'a truncated tree', windows: [], truncated: true, keyboard: notObserved }, + // The in-app placeholder can outlive the key plane during a dismissal. + { + name: 'a placeholder without keys', + windows: [ + guestNode('UITextEffectsWindow', APP_FRAME, [ + guestNode('_UIRemoteKeyboardPlaceholderView', { X: 0, Y: 509, Width: 390, Height: 335 }), + ]), + ], + truncated: false, + keyboard: notObserved, + }, + // A keyboard window that exposes no keys is not proof that the keyboard is gone. + { + name: 'a keyboard window without keys', + windows: [guestNode('UIRemoteKeyboardWindow', APP_FRAME)], + truncated: false, + keyboard: notObserved, + }, + { + name: 'a keyboard window that holds only a placeholder', + windows: [ + guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ + guestNode('_UIRemoteKeyboardPlaceholderView', { X: 0, Y: 509, Width: 390, Height: 335 }), + ]), + ], + truncated: false, + keyboard: notObserved, + }, + // A minimized or hardware keyboard leaves only its assistant bar. + { + name: 'an assistant bar without keys', + windows: [ + guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ + guestNode('TUISystemInputAssistantView', ASSISTANT_FRAME), + ]), + ], + truncated: false, + keyboard: notObserved, + }, +]; diff --git a/packages/platform-apple/src/snapshot-source/adapter.test.ts b/packages/platform-apple/src/snapshot-source/adapter.test.ts index 5921796a07..941d1ac42c 100644 --- a/packages/platform-apple/src/snapshot-source/adapter.test.ts +++ b/packages/platform-apple/src/snapshot-source/adapter.test.ts @@ -16,6 +16,7 @@ import { readSnapshotTargetProcessStartTime } from '../snapshot-process.ts'; import { DEPTH_HINT_PROBE_BACK_AFTER_USES } from './depth-hints.ts'; import { SNAPSHOT_BRIDGE_SOURCE_FILENAMES } from './cache-identity.ts'; import { MAXIMUM_REQUEST_LEVELS } from './limits.ts'; +import { ADAPTER_KEYBOARD_CASES } from './adapter-keyboard.fixtures.ts'; import { encodeSnapshotBridgeFrame, SNAPSHOT_SOURCE_PROTOCOL_VERSION, @@ -877,56 +878,11 @@ test('the Simulator AX source refuses a window whose coordinate space it cannot } }); -/** A guest node; `automationType` 19 is `Keyboard` and 20 is `Key`. */ -function guestNode( - elementType: string, - frame: Readonly<{ X: number; Y: number; Width: number; Height: number }>, - children: readonly Record[] = [], - automationType?: number, -): Record { - return { - XC_kAXXCAttributeElementType: elementType, - ...(automationType === undefined ? {} : { XC_kAXXCAttributeAutomationType: automationType }), - XC_kAXXCAttributeFrame: frame, - XC_kAXXCAttributeChildren: children, - }; -} - -const APP_FRAME = { X: 0, Y: 0, Width: 390, Height: 844 }; -const ASSISTANT_FRAME = { X: 0, Y: 509, Width: 390, Height: 44 }; -// The docked key plane stops 58 pt above the viewport bottom, as on iPhone 17 Pro. -const DOCKED_KEYBOARD_WINDOW = guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ - guestNode('TUISystemInputAssistantView', ASSISTANT_FRAME), - guestNode( - 'UIKeyboardAutomatic', - { X: 0, Y: 553, Width: 390, Height: 233 }, - [ - guestNode('UIKeyboardLayoutStar', { X: 0, Y: 553, Width: 390, Height: 233 }, [ - guestNode('UIAccessibilityElementKBKey', { X: 0, Y: 560, Width: 195, Height: 54 }, [], 20), - guestNode( - 'UIAccessibilityElementKBKey', - { X: 195, Y: 560, Width: 195, Height: 54 }, - [], - 20, - ), - ]), - ], - 19, - ), -]); - test('the Simulator AX source publishes absence only from a complete tree with no keyboard trace', async () => { const root = await mkdtempForTest('agent-device-snapshot-adapter-keyboard-'); const sourceRoot = path.join(root, 'source'); await (await import('@agent-device/host-kit/host-file')).ensureHostDirectory(sourceRoot); - for (const name of [ - 'SnapshotBridge.m', - 'SnapshotBridgeRuntime.m', - 'SnapshotBridgeRuntime.h', - 'SnapshotBridgeCapture.h', - 'SnapshotBridgeCapture.m', - 'ForegroundApplicationGeometry.h', - ]) { + for (const name of SNAPSHOT_BRIDGE_SOURCE_FILENAMES) { await writeFile(path.join(sourceRoot, name), 'native source'); } const fixture = createAdapterHost(); @@ -936,70 +892,9 @@ test('the Simulator AX source publishes absence only from a complete tree with n cacheRoot: path.join(root, 'cache'), }); const target = { ...targetForTest(), generation: 'generation-1', targetId: 'target-1' }; - const notObserved = { kind: 'unmeasurable', reason: 'keyboard-not-observed' }; - const cases = [ - { - name: 'a docked keyboard', - windows: [DOCKED_KEYBOARD_WINDOW], - truncated: false, - keyboard: { - kind: 'derived', - derivation: 'docked-tree', - frame: { x: 0, y: 553, width: 390, height: 291 }, - treeBound: true, - }, - }, - { - name: 'a complete tree', - windows: [], - truncated: false, - keyboard: { kind: 'absent', treeBound: true }, - }, - // A cut can drop the keyboard window, so a truncated tree proves only presence. - { name: 'a truncated tree', windows: [], truncated: true, keyboard: notObserved }, - // The in-app placeholder can outlive the key plane during a dismissal. - { - name: 'a placeholder without keys', - windows: [ - guestNode('UITextEffectsWindow', APP_FRAME, [ - guestNode('_UIRemoteKeyboardPlaceholderView', { X: 0, Y: 509, Width: 390, Height: 335 }), - ]), - ], - truncated: false, - keyboard: notObserved, - }, - // A keyboard window that exposes no keys is not proof that the keyboard is gone. - { - name: 'a keyboard window without keys', - windows: [guestNode('UIRemoteKeyboardWindow', APP_FRAME)], - truncated: false, - keyboard: notObserved, - }, - { - name: 'a keyboard window that holds only a placeholder', - windows: [ - guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ - guestNode('_UIRemoteKeyboardPlaceholderView', { X: 0, Y: 509, Width: 390, Height: 335 }), - ]), - ], - truncated: false, - keyboard: notObserved, - }, - // A minimized or hardware keyboard leaves only its assistant bar. - { - name: 'an assistant bar without keys', - windows: [ - guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ - guestNode('TUISystemInputAssistantView', ASSISTANT_FRAME), - ]), - ], - truncated: false, - keyboard: notObserved, - }, - ]; try { - for (const entry of cases) { + for (const entry of ADAPTER_KEYBOARD_CASES) { fixture.windows = entry.windows; fixture.truncated = entry.truncated; for (const acquisitionIntent of ['full', 'surface-observation'] as const) { From 05c8da97675631f7f068024fd342df91e3855988 Mon Sep 17 00:00:00 2001 From: DongYun Kang Date: Wed, 4 Nov 2026 22:29:22 +0900 Subject: [PATCH 4/4] docs(adr): leave ADR 0004 unchanged in the bridge keyboard fact change The bridge fact contradicts no statement in ADR 0004, and the commands docs, the changeset and the code comments already record the behavior. The provider-producer sentence that #4207 made stale is outside this change. Co-Authored-By: Claude Opus 5.5 --- .../adr/0004-ios-snapshot-backend-strategy.md | 34 ++----------------- 1 file changed, 3 insertions(+), 31 deletions(-) diff --git a/docs/adr/0004-ios-snapshot-backend-strategy.md b/docs/adr/0004-ios-snapshot-backend-strategy.md index ae0d995ab5..3362b95fd1 100644 --- a/docs/adr/0004-ios-snapshot-backend-strategy.md +++ b/docs/adr/0004-ios-snapshot-backend-strategy.md @@ -623,9 +623,9 @@ Thus, the pass returns that tree exactly as reported. This is a structural consequence of the same rule applied uniformly. It is not the per-tier omission that a hand-threaded space would have made it. -The provider producers (`appium-source`, `limrun-ios-tree`, `revyl-ios-tree`) never see the app's windows either. -Their keyboard fact comes only from the docked-tree derivation or from a separate visibility probe, and it never states absence. -A consumer reads a missing or `unmeasurable` fact as "this producer did not measure". +The provider producers (`appium-source`, `limrun-ios-tree`) never see the app's windows either. +A capture from any of them publishes no keyboard fact. +A consumer reads that silence as "this producer did not measure". Those paths publish what the platform reported. Thus, the last reader that can still refuse geometry that it cannot place is the tap-path keyboard guard, and its width rule therefore remains. The rule detects un-normalized arrival, not a standing fact about iOS. @@ -777,31 +777,3 @@ Rejected alternatives: - Move the tree by the window origin. The bridge would then be the only correct producer for a windowed app. The keyboard window and alerts were not measured in that mode, so the refusal stays until they are. - Read the screen size on the host from the device type profile or from `simctl io`. That adds a host process and a format that Apple does not document. It also does not give the window origin. - -## Amendment: the bridge's keyboard fact (follow-up to PR #4207) - -The bridge reads the app's root with the same remote AX snapshot API as the runner. -In that tree, the system keyboard window (`UIRemoteKeyboardWindow`) is one of the app's own windows, and its frames are in app space. -The bridge sets `truncated` on every cut that it makes. - -Decision. A bridge acquisition publishes a keyboard fact (`observeCompleteTreeKeyboard`) for both intents. -A docked keyboard in the tree publishes the docked-tree band (`derived`, `treeBound`). -A complete tree (`truncated: false`) with no keyboard trace publishes `absent` with `treeBound`. -A keyboard trace is the keyboard window (`UIRemoteKeyboardWindow`), a `Keyboard` or `Key` node, or a node with an iOS keyboard role, which includes `_UIRemoteKeyboardPlaceholderView` and `TUISystemInputAssistantView`. -A truncated tree, or a complete tree that holds only a trace, publishes `unmeasurable`. -Thus, a keyboard window that exposes no keys, for example a hardware keyboard that shows only the assistant bar, never publishes absence. - -Reason. Post-gesture stabilization counts a keyboard change as fresh only between two tree-bound facts. -Without a bridge fact, a drag that only dismissed the keyboard waited for the distrust cap (about 3.5 s) on the default Simulator route. - -Rejected alternatives: - -- Publish absence from every tree without a `Keyboard` or `Key` node. A minimized keyboard, a placeholder that outlives the key plane during a dismissal, and an out-of-process keyboard can leave only a trace. Absence there could end the distrust window on a stale read. -- Publish absence from a truncated tree. A cut can drop the keyboard window. - -Consequences. `snapshot --json` shows `keyboard` for bridge captures. -On a complete tree with no keyboard trace, the tap guard reads `no-keyboard` instead of `undetermined`. -Only `occluded` changes a tap, so taps do not change. -The bridge band excludes the input assistant bar, as the band that the tap guard derived for bridge captures already did. -The runner's measured band includes that bar. -Open gaps: third-party keyboards, iPad floating and split keyboards, and hardware-keyboard states were not captured on the bridge.