diff --git a/.changeset/ios-ax-bridge-keyboard-fact.md b/.changeset/ios-ax-bridge-keyboard-fact.md new file mode 100644 index 0000000000..2521f5d228 --- /dev/null +++ b/.changeset/ios-ax-bridge-keyboard-fact.md @@ -0,0 +1,5 @@ +--- +'agent-device': patch +--- + +Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree, or a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar), stays `unmeasurable`. diff --git a/contracts/fixtures/tap-keyboard-occlusion-policy.json b/contracts/fixtures/tap-keyboard-occlusion-policy.json index 47450c03ad..9b63b502a9 100644 --- a/contracts/fixtures/tap-keyboard-occlusion-policy.json +++ b/contracts/fixtures/tap-keyboard-occlusion-policy.json @@ -1,5 +1,5 @@ { - "description": "Golden table for keyboard occlusion on interaction tap points (#2589), consumed by packages/contracts/src/tap-keyboard-occlusion.ts. A visible software keyboard is its own system surface: it is never a covering sibling of app content, so the same-window occlusion classifier cannot see it, and it leaves the app window rect intact, so the viewport rule passes a covered tab bar. The band runs from the topmost keyboard node down to the bottom of the viewport, across the keyboard's own columns; the tap point is the point the interaction dispatches: the rect center the platform aims at for a ref handed over by name, or the touch point a parent with interactive descendants resolves instead. `nodes` are captured snapshot nodes as they appear in the projection the interaction resolved against \u2014 note that regular and interactive projections prune the keyboard's own container, leaving Key nodes beside app chrome. Three rules decide whether reported geometry may be measured at all. Docking: the band is derived only where the keyboard's own reported geometry reaches the viewport's bottom edge within 80 units \u2014 measured on iPhone 17 Pro (26.2) the reported key plane stops 58 pt short of an 874 pt portrait viewport and lands exactly on a 402 pt landscape one, and on iPad Pro 11-inch (M4) 3 pt short of a 1210 pt viewport \u2014 so an iPad floating keyboard, an app-drawn keypad, or an IME that stops above its own navigation bar stays unmeasured and fails open. The budget is counted in the same units as the rects it guards, which is what a measured Android IME asks for rather than what a dp table would give: with three-button navigation on Pixel 7 (Android 16, 1080x2400 at 420 dpi), Gboard's reported region runs to the physical display bottom at 2400 while the 126 px navigation bar occupies 2274..2400 inside it, so no Android keyboard measured so far needs slack for the bar. Width: geometry that comes back taller than it is wide did not arrive in the app's orientation space, i.e. it came from a capture that normalized no space \u2014 the producers that can name the app's interface orientation either turn it back (the runner's tree tiers) or refuse the capture (the Simulator AX bridge) \u2014 and is refused measurement for the same reason. Tiling: the keys themselves must form one unbroken run of columns, because a split iPad keyboard docks both clusters at the bottom edge while its surface container still spans the whole screen, and the app content between the clusters is visible and tappable. Reported seams are zero on every measured tree \u2014 iPhone 17 Pro's reported row adjoins across 395 pt, iPad Pro 11-inch's across 743.5 pt, and Gboard's across 1070 px \u2014 so the allowance covers rounding rather than layout, and a gapped band is refused measurement like the two above. That rule is deliberately blunt in both directions, and the last two cases name the trade: a sparse layout such as the iOS emoji panel reports corner keys over a wide container, does not tile, and reads `undetermined` \u2014 the guard abstains there rather than inventing which gaps are app content; and a projection that reports the container with no keys at all has nothing left to check, so the band is the container's own claim. Every captured iOS and Android keyboard measured so far reports its keys, so the second shape is the harness's, not a captured one. `target` names either a resolved node index or a bare coordinate point; both paths read one decision and choose their own consequence. A case that reaches a band declares its whole shape \u2014 the frame the refusal and the disclosure both quote, and the control rects that excuse a bare coordinate, which is where plane exclusion shows up. Provider facts distinguish measured visible frames, derived docked-tree bands, visibility-only probes, explicit absence, and unknown. Missing keyboard nodes are undetermined, never absence. A carried derived band was established before projection and stays usable after projection removes keyboard nodes; it obeys the same docking/orientation/tiling rules at acquisition. A measured floating frame needs no bottom extension.", + "description": "Golden table for keyboard occlusion on interaction tap points (#2589), consumed by packages/contracts/src/tap-keyboard-occlusion.ts. A visible software keyboard is its own system surface: it is never a covering sibling of app content, so the same-window occlusion classifier cannot see it, and it leaves the app window rect intact, so the viewport rule passes a covered tab bar. The band runs from the topmost keyboard node down to the bottom of the viewport, across the keyboard's own columns; the tap point is the point the interaction dispatches: the rect center the platform aims at for a ref handed over by name, or the touch point a parent with interactive descendants resolves instead. `nodes` are captured snapshot nodes as they appear in the projection the interaction resolved against \u2014 note that regular and interactive projections prune the keyboard's own container, leaving Key nodes beside app chrome. Three rules decide whether reported geometry may be measured at all. Docking: the band is derived only where the keyboard's own reported geometry reaches the viewport's bottom edge within 80 units \u2014 measured on iPhone 17 Pro (26.2) the reported key plane stops 58 pt short of an 874 pt portrait viewport and lands exactly on a 402 pt landscape one, and on iPad Pro 11-inch (M4) 3 pt short of a 1210 pt viewport \u2014 so an iPad floating keyboard, an app-drawn keypad, or an IME that stops above its own navigation bar stays unmeasured and fails open. The budget is counted in the same units as the rects it guards, which is what a measured Android IME asks for rather than what a dp table would give: with three-button navigation on Pixel 7 (Android 16, 1080x2400 at 420 dpi), Gboard's reported region runs to the physical display bottom at 2400 while the 126 px navigation bar occupies 2274..2400 inside it, so no Android keyboard measured so far needs slack for the bar. Width: geometry that comes back taller than it is wide did not arrive in the app's orientation space, i.e. it came from a capture that normalized no space \u2014 the producers that can name the app's interface orientation either turn it back (the runner's tree tiers) or refuse the capture (the Simulator AX bridge) \u2014 and is refused measurement for the same reason. Tiling: the keys themselves must form one unbroken run of columns, because a split iPad keyboard docks both clusters at the bottom edge while its surface container still spans the whole screen, and the app content between the clusters is visible and tappable. Reported seams are zero on every measured tree \u2014 iPhone 17 Pro's reported row adjoins across 395 pt, iPad Pro 11-inch's across 743.5 pt, and Gboard's across 1070 px \u2014 so the allowance covers rounding rather than layout, and a gapped band is refused measurement like the two above. That rule is deliberately blunt in both directions, and the last two cases name the trade: a sparse layout such as the iOS emoji panel reports corner keys over a wide container, does not tile, and reads `undetermined` \u2014 the guard abstains there rather than inventing which gaps are app content; and a projection that reports the container with no keys at all has nothing left to check, so the band is the container's own claim. Every captured iOS and Android keyboard measured so far reports its keys, so the second shape is the harness's, not a captured one. `target` names either a resolved node index or a bare coordinate point; both paths read one decision and choose their own consequence. A case that reaches a band declares its whole shape \u2014 the frame the refusal and the disclosure both quote, and the control rects that excuse a bare coordinate, which is where plane exclusion shows up. Provider facts distinguish measured visible frames, derived docked-tree bands, visibility-only probes, explicit absence, and unknown. Missing keyboard nodes are undetermined, never absence. The one exception is a capture of the app's whole in-process tree, such as the Simulator AX bridge, where the system keyboard is one of the app's own windows: `treeComplete` pins that observation, and only a complete tree with no keyboard trace at all (no keyboard window, no `Keyboard` or `Key` node, and no keyboard role, which includes the remote keyboard placeholder and the input assistant) establishes absence; a keyboard window that exposes no keys stays undetermined. A carried derived band was established before projection and stays usable after projection removes keyboard nodes; it obeys the same docking/orientation/tiling rules at acquisition. A measured floating frame needs no bottom extension.", "constants": { "occlusionReason": "tap_keyboard_occludes_target" }, @@ -5304,6 +5304,668 @@ } ] } + }, + { + "name": "a complete in-process tree with no keyboard trace establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 1 + }, + "keyboard": { + "kind": "absent", + "treeBound": true + }, + "treeComplete": true, + "expected": { + "kind": "no-keyboard" + } + }, + { + "name": "a truncated in-process tree with no keyboard nodes never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 1 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": false, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree holding only the remote keyboard placeholder never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Other", + "role": "_UIRemoteKeyboardPlaceholderView", + "identifier": "inputView", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 539, + "width": 402, + "height": 335 + } + }, + { + "index": 2, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 2 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree holding only the input assistant bar never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Other", + "role": "TUISystemInputAssistantView", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 830, + "width": 402, + "height": 44 + } + }, + { + "index": 2, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 2 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree holding a keyboard window without keys never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Window", + "role": "UIRemoteKeyboardWindow", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 2, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 2 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree holding a keyboard window with only the input assistant bar never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Window", + "role": "UIRemoteKeyboardWindow", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 2, + "type": "Other", + "role": "TUISystemInputAssistantView", + "parentIndex": 1, + "rect": { + "x": 0, + "y": 830, + "width": 402, + "height": 44 + } + }, + { + "index": 3, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 3 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree holding a keyboard window with only the remote keyboard placeholder never establishes absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "Window", + "role": "UIRemoteKeyboardWindow", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 2, + "type": "Other", + "role": "_UIRemoteKeyboardPlaceholderView", + "parentIndex": 1, + "rect": { + "x": 0, + "y": 539, + "width": 402, + "height": 335 + } + }, + { + "index": 3, + "type": "Button", + "label": "Form", + "parentIndex": 0, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + } + ], + "target": { + "index": 3 + }, + "keyboard": { + "kind": "unmeasurable", + "reason": "keyboard-not-observed" + }, + "treeComplete": true, + "expected": { + "kind": "undetermined" + } + }, + { + "name": "a complete in-process tree with a docked keyboard carries its derived band, not absence", + "viewport": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + }, + "nodes": [ + { + "index": 0, + "type": "Application", + "rect": { + "x": 0, + "y": 0, + "width": 402, + "height": 874 + } + }, + { + "index": 1, + "type": "TabBar", + "label": "Tab Bar", + "parentIndex": 0, + "rect": { + "x": 0, + "y": 791, + "width": 402, + "height": 83 + } + }, + { + "index": 2, + "type": "Button", + "label": "Form", + "parentIndex": 1, + "rect": { + "x": 162, + "y": 795, + "width": 74, + "height": 54 + } + }, + { + "index": 3, + "type": "Key", + "label": "Q", + "parentIndex": 0, + "rect": { + "x": 4.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 4, + "type": "Key", + "label": "W", + "parentIndex": 0, + "rect": { + "x": 44, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 5, + "type": "Key", + "label": "E", + "parentIndex": 0, + "rect": { + "x": 83.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 6, + "type": "Key", + "label": "R", + "parentIndex": 0, + "rect": { + "x": 123, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 7, + "type": "Key", + "label": "T", + "parentIndex": 0, + "rect": { + "x": 162.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 8, + "type": "Key", + "label": "Y", + "parentIndex": 0, + "rect": { + "x": 202, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 9, + "type": "Key", + "label": "U", + "parentIndex": 0, + "rect": { + "x": 241.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 10, + "type": "Key", + "label": "I", + "parentIndex": 0, + "rect": { + "x": 281, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 11, + "type": "Key", + "label": "O", + "parentIndex": 0, + "rect": { + "x": 320.67, + "y": 590, + "width": 39.33, + "height": 54 + } + }, + { + "index": 12, + "type": "Key", + "label": "P", + "parentIndex": 0, + "rect": { + "x": 360, + "y": 590, + "width": 39.67, + "height": 54 + } + }, + { + "index": 13, + "type": "Key", + "label": "return", + "parentIndex": 0, + "rect": { + "x": 300, + "y": 752, + "width": 99, + "height": 54 + } + } + ], + "target": { + "index": 2 + }, + "keyboard": { + "kind": "derived", + "derivation": "docked-tree", + "frame": { + "x": 4.67, + "y": 590, + "width": 395, + "height": 284 + }, + "treeBound": true + }, + "treeComplete": true, + "expected": { + "kind": "occluded", + "frame": { + "x": 4.67, + "y": 590, + "width": 395, + "height": 284 + }, + "controlRects": [ + { + "x": 4.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 44, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 83.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 123, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 162.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 202, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 241.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 281, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 320.67, + "y": 590, + "width": 39.33, + "height": 54 + }, + { + "x": 360, + "y": 590, + "width": 39.67, + "height": 54 + }, + { + "x": 300, + "y": 752, + "width": 99, + "height": 54 + } + ] + } } ] } diff --git a/packages/contracts/src/tap-keyboard-occlusion.test.ts b/packages/contracts/src/tap-keyboard-occlusion.test.ts index ce5c07b4dd..22af08f474 100644 --- a/packages/contracts/src/tap-keyboard-occlusion.test.ts +++ b/packages/contracts/src/tap-keyboard-occlusion.test.ts @@ -9,6 +9,7 @@ import type { } from '@agent-device/kernel/snapshot'; import { acquireProviderKeyboard, + observeCompleteTreeKeyboard, resolveKeyboardTapOcclusion, TAP_KEYBOARD_OCCLUDES_TARGET_DETAILS, TAP_KEYBOARD_OCCLUDES_TARGET_REASON, @@ -34,6 +35,8 @@ type FixtureCase = { target: { index: number } | { point: { x: number; y: number } }; keyboard?: SnapshotKeyboardBandFact; observationVisibility?: boolean; + /** Whether a producer that reads the app's whole in-process tree captured `nodes` uncut. */ + treeComplete?: boolean; expected: { kind: string; frame?: Rect; controlRects?: readonly Rect[] }; }; @@ -84,6 +87,17 @@ test('keyboard tap occlusion agrees with every golden parity table case', () => `${fixture.name}: acquisition observation`, ); } + if (fixture.treeComplete !== undefined) { + assert.deepEqual( + observeCompleteTreeKeyboard({ + nodes, + viewport: fixture.viewport, + complete: fixture.treeComplete, + }), + fixture.keyboard, + `${fixture.name}: complete-tree observation`, + ); + } const occlusion = resolveKeyboardTapOcclusion({ nodes, viewport: fixture.viewport, diff --git a/packages/contracts/src/tap-keyboard-occlusion.ts b/packages/contracts/src/tap-keyboard-occlusion.ts index f332649468..3202e01823 100644 --- a/packages/contracts/src/tap-keyboard-occlusion.ts +++ b/packages/contracts/src/tap-keyboard-occlusion.ts @@ -66,6 +66,8 @@ export const TAP_KEYBOARD_OCCLUDES_TARGET_DETAILS = Object.freeze({ }); const IOS_KEYBOARD_TYPE_NAMES: ReadonlySet = new Set(['keyboard', 'key']); +/** The system window that hosts the remote keyboard, with or without keys in it. */ +const IOS_KEYBOARD_WINDOW_ROLE = 'UIRemoteKeyboardWindow'; /** The occluding band of the visible keyboard, plus the rects that prove a tap meant the keyboard. */ export type KeyboardSurface = { @@ -356,6 +358,31 @@ export function acquireProviderKeyboard(params: ProviderKeyboardInput): { }; } +/** + * The keyboard fact of a producer that reads the target app's whole in-process tree in one capture, + * as the Simulator AX bridge does: the keyboard window is one of the app's own windows there, so a + * complete tree with no keyboard window and no keyboard node proves absence the way a full XCTest scan + * does. A truncated tree, or a keyboard window that exposes no keys, proves only presence. + */ +export function observeCompleteTreeKeyboard( + params: Readonly<{ nodes: readonly RawSnapshotNode[]; viewport: Rect | null; complete: boolean }>, +): SnapshotKeyboardBandFact { + const fact = observeProviderKeyboard(params); + if (fact.kind !== 'unmeasurable' || !params.complete) return fact; + return params.nodes.some(isKeyboardTraceNode) ? fact : { kind: 'absent', treeBound: true }; +} + +/** Any node the keyboard leaves in the tree, including its window and the in-app placeholder. */ +function isKeyboardTraceNode(node: RawSnapshotNode): boolean { + return ( + node.role === IOS_KEYBOARD_WINDOW_ROLE || + isKeyboardSurfaceNode(node) || + hasIosKeyboardRole(node, IOS_KEYBOARD_SETTLE_CHROME_ROLES) || + hasIosKeyboardRole(node, IOS_KEYBOARD_SUBTREE_CHROME_ROLES) || + hasIosKeyboardRole(node, IOS_KEYBOARD_CHROME_MARKER_ROLES) + ); +} + function observeProviderKeyboard(params: ProviderKeyboardInput): SnapshotKeyboardBandFact { const frame = deriveDockedKeyboardFrame(params.nodes, params.viewport); // Derived from the nodes this capture returns, so a stale tree derives the stale tree's band. diff --git a/packages/daemon/src/core/__tests__/post-gesture-stabilization.test.ts b/packages/daemon/src/core/__tests__/post-gesture-stabilization.test.ts index 3d774a15a1..727bdfec75 100644 --- a/packages/daemon/src/core/__tests__/post-gesture-stabilization.test.ts +++ b/packages/daemon/src/core/__tests__/post-gesture-stabilization.test.ts @@ -1,5 +1,8 @@ import type { CommandFlags } from '@agent-device/contracts/command'; -import { acquireProviderKeyboard } from '@agent-device/contracts/tap-keyboard-occlusion'; +import { + acquireProviderKeyboard, + observeCompleteTreeKeyboard, +} from '@agent-device/contracts/tap-keyboard-occlusion'; import assert from 'node:assert/strict'; import { afterEach, test, vi } from 'vitest'; import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures'; @@ -608,6 +611,84 @@ test('capturePostGestureStabilizedResult still distrusts a stale WebDriver sourc assert.equal(settled, 0); }); +/** A Simulator AX bridge capture: one complete read of the app's tree, keyboard window included. */ +function bridgeCapture(keyboard: boolean) { + const source = makeSnapshotState([ + ...pickupSnapshot(500).nodes, + ...(keyboard ? keyboardWindowNodes() : []), + ]); + return { + ...source, + keyboard: observeCompleteTreeKeyboard({ + nodes: source.nodes, + viewport: { x: 0, y: 0, width: 390, height: 844 }, + complete: true, + }), + }; +} + +async function stabilizeBridgeCaptures(params: { before: boolean; after: boolean }) { + vi.useFakeTimers(); + const session = makeSession('ios'); + session.snapshot = bridgeCapture(params.before); + markPostGestureStabilization(session, 'swipe'); + const baselineKeyboard = session.snapshot.keyboard; + const capture = vi.fn(async () => bridgeCapture(params.after)); + + const resultPromise = withDiagnosticsScope({}, async () => { + const result = await capturePostGestureStabilizedResult({ + session, + capture, + readSnapshot: (snapshot) => snapshot, + }); + return { + result, + staleAccepts: countDiagnosticEventsByPhase(['post_gesture_snapshot_stale_accept']), + settled: countDiagnosticEventsByPhase(['post_gesture_snapshot_stabilized']), + }; + }); + + await vi.advanceTimersByTimeAsync(10_000); + const { result, staleAccepts, settled } = await resultPromise; + return { + staleAccepts, + settled, + baselineKeyboard, + quietKeyboard: result.value.keyboard, + postGestureOutcome: result.postGestureOutcome, + captures: capture.mock.calls.length, + }; +} + +test('capturePostGestureStabilizedResult settles when a complete bridge tree loses the keyboard it derived (iOS)', async () => { + const outcome = await stabilizeBridgeCaptures({ before: true, after: false }); + + assert.deepEqual(outcome.baselineKeyboard, { + kind: 'derived', + derivation: 'docked-tree', + frame: { x: 0, y: 500, width: 390, height: 344 }, + treeBound: true, + }); + assert.deepEqual(outcome.quietKeyboard, { kind: 'absent', treeBound: true }); + assert.equal(outcome.settled, 1); + assert.equal(outcome.staleAccepts, 0); + assert.equal(outcome.postGestureOutcome, undefined); + assert.equal(outcome.captures, 2); +}); + +test('capturePostGestureStabilizedResult still distrusts bridge trees whose keyboard fact did not change (iOS)', async () => { + // The same fact on both sides proves nothing, so the content comparison decides, and it never moved. + for (const keyboard of [false, true]) { + const outcome = await stabilizeBridgeCaptures({ before: keyboard, after: keyboard }); + + assert.equal(outcome.quietKeyboard?.kind, keyboard ? 'derived' : 'absent'); + assert.deepEqual(outcome.quietKeyboard, outcome.baselineKeyboard); + assert.equal(outcome.staleAccepts, 1, `keyboard ${keyboard}`); + assert.equal(outcome.settled, 0, `keyboard ${keyboard}`); + vi.useRealTimers(); + } +}); + // --- #1569: a backend swap mid-poll is not comparable evidence --- test('capturePostGestureStabilizedResult re-baselines instead of concluding when the backend changes (iOS)', async () => { diff --git a/packages/platform-apple/src/snapshot-source/adapter-keyboard.fixtures.ts b/packages/platform-apple/src/snapshot-source/adapter-keyboard.fixtures.ts new file mode 100644 index 0000000000..2ed559dbf0 --- /dev/null +++ b/packages/platform-apple/src/snapshot-source/adapter-keyboard.fixtures.ts @@ -0,0 +1,110 @@ +/** + * Guest trees for the Simulator AX source's keyboard fact: each case names the windows the fake + * guest reports under its root, whether it cut the tree, and the fact the capture must publish. + */ + +/** A guest node; `automationType` 19 is `Keyboard` and 20 is `Key`. */ +function guestNode( + elementType: string, + frame: Readonly<{ X: number; Y: number; Width: number; Height: number }>, + children: readonly Record[] = [], + automationType?: number, +): Record { + return { + XC_kAXXCAttributeElementType: elementType, + ...(automationType === undefined ? {} : { XC_kAXXCAttributeAutomationType: automationType }), + XC_kAXXCAttributeFrame: frame, + XC_kAXXCAttributeChildren: children, + }; +} + +const APP_FRAME = { X: 0, Y: 0, Width: 390, Height: 844 }; +const ASSISTANT_FRAME = { X: 0, Y: 509, Width: 390, Height: 44 }; +// The docked key plane stops 58 pt above the viewport bottom, as on iPhone 17 Pro. +const DOCKED_KEYBOARD_WINDOW = guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ + guestNode('TUISystemInputAssistantView', ASSISTANT_FRAME), + guestNode( + 'UIKeyboardAutomatic', + { X: 0, Y: 553, Width: 390, Height: 233 }, + [ + guestNode('UIKeyboardLayoutStar', { X: 0, Y: 553, Width: 390, Height: 233 }, [ + guestNode('UIAccessibilityElementKBKey', { X: 0, Y: 560, Width: 195, Height: 54 }, [], 20), + guestNode( + 'UIAccessibilityElementKBKey', + { X: 195, Y: 560, Width: 195, Height: 54 }, + [], + 20, + ), + ]), + ], + 19, + ), +]); + +const notObserved = { kind: 'unmeasurable', reason: 'keyboard-not-observed' }; + +export const ADAPTER_KEYBOARD_CASES: readonly Readonly<{ + name: string; + windows: readonly Record[]; + truncated: boolean; + keyboard: Record; +}>[] = [ + { + name: 'a docked keyboard', + windows: [DOCKED_KEYBOARD_WINDOW], + truncated: false, + keyboard: { + kind: 'derived', + derivation: 'docked-tree', + frame: { x: 0, y: 553, width: 390, height: 291 }, + treeBound: true, + }, + }, + { + name: 'a complete tree', + windows: [], + truncated: false, + keyboard: { kind: 'absent', treeBound: true }, + }, + // A cut can drop the keyboard window, so a truncated tree proves only presence. + { name: 'a truncated tree', windows: [], truncated: true, keyboard: notObserved }, + // The in-app placeholder can outlive the key plane during a dismissal. + { + name: 'a placeholder without keys', + windows: [ + guestNode('UITextEffectsWindow', APP_FRAME, [ + guestNode('_UIRemoteKeyboardPlaceholderView', { X: 0, Y: 509, Width: 390, Height: 335 }), + ]), + ], + truncated: false, + keyboard: notObserved, + }, + // A keyboard window that exposes no keys is not proof that the keyboard is gone. + { + name: 'a keyboard window without keys', + windows: [guestNode('UIRemoteKeyboardWindow', APP_FRAME)], + truncated: false, + keyboard: notObserved, + }, + { + name: 'a keyboard window that holds only a placeholder', + windows: [ + guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ + guestNode('_UIRemoteKeyboardPlaceholderView', { X: 0, Y: 509, Width: 390, Height: 335 }), + ]), + ], + truncated: false, + keyboard: notObserved, + }, + // A minimized or hardware keyboard leaves only its assistant bar. + { + name: 'an assistant bar without keys', + windows: [ + guestNode('UIRemoteKeyboardWindow', APP_FRAME, [ + guestNode('TUISystemInputAssistantView', ASSISTANT_FRAME), + ]), + ], + truncated: false, + keyboard: notObserved, + }, +]; diff --git a/packages/platform-apple/src/snapshot-source/adapter.test.ts b/packages/platform-apple/src/snapshot-source/adapter.test.ts index 7f4ecfa446..941d1ac42c 100644 --- a/packages/platform-apple/src/snapshot-source/adapter.test.ts +++ b/packages/platform-apple/src/snapshot-source/adapter.test.ts @@ -16,6 +16,7 @@ import { readSnapshotTargetProcessStartTime } from '../snapshot-process.ts'; import { DEPTH_HINT_PROBE_BACK_AFTER_USES } from './depth-hints.ts'; import { SNAPSHOT_BRIDGE_SOURCE_FILENAMES } from './cache-identity.ts'; import { MAXIMUM_REQUEST_LEVELS } from './limits.ts'; +import { ADAPTER_KEYBOARD_CASES } from './adapter-keyboard.fixtures.ts'; import { encodeSnapshotBridgeFrame, SNAPSHOT_SOURCE_PROTOCOL_VERSION, @@ -554,6 +555,8 @@ type AdapterFixture = { runningSurfaceHost: boolean; /** The surface host check each request carried: its executables and Simulator, or undefined. */ surfaceHostChecks: Array | undefined>; + /** The windows the fake guest reports under its root when no other tree shape is selected. */ + windows: readonly Record[]; omitRecovery: boolean; diagnostics: Record[]; startedTargets: Array[0]>; @@ -591,6 +594,7 @@ function createAdapterHost(buildDelayMs = 0): AdapterFixture { omitViewport: false, runningSurfaceHost: false, surfaceHostChecks: [], + windows: [], omitRecovery: false, diagnostics: [], startedTargets: [], @@ -813,7 +817,7 @@ class AdapterSocket extends EventEmitter implements SnapshotSourceSocket { ], }, ] - : [], + : this.fixture.windows, }, }, { @@ -873,3 +877,43 @@ test('the Simulator AX source refuses a window whose coordinate space it cannot await rm(root, { recursive: true, force: true }); } }); + +test('the Simulator AX source publishes absence only from a complete tree with no keyboard trace', async () => { + const root = await mkdtempForTest('agent-device-snapshot-adapter-keyboard-'); + const sourceRoot = path.join(root, 'source'); + await (await import('@agent-device/host-kit/host-file')).ensureHostDirectory(sourceRoot); + for (const name of SNAPSHOT_BRIDGE_SOURCE_FILENAMES) { + await writeFile(path.join(sourceRoot, name), 'native source'); + } + const fixture = createAdapterHost(); + const source = createSimulatorSnapshotSource({ + host: fixture.host, + sourceRoot, + cacheRoot: path.join(root, 'cache'), + }); + const target = { ...targetForTest(), generation: 'generation-1', targetId: 'target-1' }; + + try { + for (const entry of ADAPTER_KEYBOARD_CASES) { + fixture.windows = entry.windows; + fixture.truncated = entry.truncated; + for (const acquisitionIntent of ['full', 'surface-observation'] as const) { + const outcome = await source.acquire({ + target, + hint: deriveIosCaptureHint(createIosSnapshotRequest({ acquisitionIntent })), + }); + assert.equal(outcome.stage, 'acquired', `${entry.name} (${acquisitionIntent})`); + if (outcome.stage !== 'acquired') continue; + assert.equal(outcome.acquisition.truncated, entry.truncated); + assert.deepEqual( + outcome.acquisition.keyboard, + entry.keyboard, + `${entry.name} (${acquisitionIntent})`, + ); + } + } + } finally { + await source.close(); + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/packages/platform-apple/src/snapshot-source/adapter.ts b/packages/platform-apple/src/snapshot-source/adapter.ts index a62ac5ea1e..9540a7acf3 100644 --- a/packages/platform-apple/src/snapshot-source/adapter.ts +++ b/packages/platform-apple/src/snapshot-source/adapter.ts @@ -1,4 +1,5 @@ import { AppError } from '@agent-device/kernel/errors'; +import { observeCompleteTreeKeyboard } from '@agent-device/contracts/tap-keyboard-occlusion'; import type { CaptureHint, IosSnapshotAcquisition, @@ -246,6 +247,11 @@ function createAcquisition( nodes, truncated, viewport: decoded.viewport, + keyboard: observeCompleteTreeKeyboard({ + nodes, + viewport: decoded.viewport.kind === 'reported' ? decoded.viewport.rect : null, + complete: !truncated, + }), lineage, residue, }; diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index ecce60609b..29b9116c00 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -607,7 +607,7 @@ agent-device is text 'id="greeting"' "Welcome back" - On the web, on-screen text the browser cannot hand an action to (static text, a `status` announcement) is in the tree and readable by `wait text`, `find text`, `get text`, and `text=`, but carries no `@ref` and has no rect: `find ... list` prints it as `(no ref)`, and it is never a tap target. Such text is published on `value`, so `value="X"` can match page text as well as a field holding that text, and a predicate that needs one element then reports `AMBIGUOUS_MATCH`; use `text=` for page text, or add `editable=true` to name the field. - The first read after `scroll`, `swipe`, or `gesture swipe` waits for two captures to agree. When that proves something about the gesture, `is`, `get`, `find`, `wait`, interactions that captured it, and `snapshot` add a warning, and all but `snapshot` report `postGestureOutcome` (`{ "kind", "gesture": { "action", "positionals" } }`) in `data` or `error.details`: - `kind: "unsettled"`: the screen was still changing. A miss is not proof of absence; read again. `is absent` refuses with `observation: "unsettled"`, and `wait absent` keeps polling. - - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner does, a WebDriver (Appium) visibility probe does not. Otherwise only the content is compared. + - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window. Otherwise only the content is compared. ## Replay