Repository navigation
fix(ios): give simulator AX bridge captures a tree-bound keyboard fact - #4217
Conversation
The default iOS Simulator route (`simulator-ax-bridge`) published no keyboard fact. Post-gesture stabilization counts a keyboard change only between two tree-bound facts, so a drag that only dismissed the keyboard still waited for the distrust cap (about 3.7 s) on that route. The bridge reads the app's root with the same AX snapshot API as the runner, and the system keyboard window is one of the app's own windows in that tree. The adapter now publishes `observeCompleteTreeKeyboard`: the docked-tree band when the tree holds a docked keyboard, and `absent` with `treeBound` only when the tree is complete and holds no keyboard trace (no `Keyboard`/`Key` node and no iOS keyboard role, which includes the remote keyboard placeholder and the input assistant). A truncated tree, or one that holds only a trace, publishes no absence. Tests pin the adapter's fact for docked, complete, truncated, placeholder-only, and assistant-only trees; add complete-tree rows to the tap keyboard occlusion table; and drive the stabilization loop with bridge-shaped `derived -> absent` (trusts) and unchanged `absent -> absent` / `derived -> derived` pairs (fall back to content). The bridge gets its own changeset, and the `no-effect` docs and ADR 0004 now name the bridge as a follow-up to the tree-bound keyboard change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A complete Simulator AX bridge tree that held a `UIRemoteKeyboardWindow` with no keys published `absent` with `treeBound`: the window class was not a keyboard trace. A keyboard window that is present but exposes no keys, for example a hardware keyboard that shows only the assistant bar, is a partial reading, so it must stay `unmeasurable`. The keyboard window now counts as a keyboard trace. Only a complete tree with no keyboard window, no `Keyboard` or `Key` node, and no keyboard role publishes absence. Adapter tests and golden rows pin a keyboard window without keys, with only the assistant bar, and with only the remote keyboard placeholder. The changeset, the `no-effect` docs, and ADR 0004 name the keyboard window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebase onto the surface-host check took adapter.test.ts past the 1000-line cap. The guest trees now live beside it, and the setup reads SNAPSHOT_BRIDGE_SOURCE_FILENAMES, so it also writes the surface-host sources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bridge fact contradicts no statement in ADR 0004, and the commands docs, the changeset and the code comments already record the behavior. The provider-producer sentence that callstack#4207 made stale is outside this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@kdy1 is attempting to deploy a commit to the Callstack Team on Vercel. A member of the Team first needs to authorize it. |
There was a problem hiding this comment.
3 issues found across 9 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/contracts/src/tap-keyboard-occlusion.ts">
<violation number="1" location="packages/contracts/src/tap-keyboard-occlusion.ts:371">
P2: This returns a `derived` fact for truncated trees whenever their partial key geometry passes the docking checks. Truncation can omit part of the keyboard, so the partial band can misclassify taps; return `unmeasurable` whenever `complete` is false.</violation>
</file>
<file name=".changeset/ios-ax-bridge-keyboard-fact.md">
<violation number="1" location=".changeset/ios-ax-bridge-keyboard-fact.md:5">
P3: The changeset overstates truncated-tree behavior: a truncated capture can publish `derived` or `tree-rejected`, not only `unmeasurable`. Describe truncation as never proving absence, while preserving the fact established by its captured evidence.</violation>
</file>
<file name="website/docs/docs/commands.md">
<violation number="1" location="website/docs/docs/commands.md:610">
P3: The command documentation narrows the bridge’s absence rule too far. Mention `Keyboard`/`Key` nodes and iOS keyboard roles, because missing only the keyboard window does not prove dismissal.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
| params: Readonly<{ nodes: readonly RawSnapshotNode[]; viewport: Rect | null; complete: boolean }>, | ||
| ): SnapshotKeyboardBandFact { | ||
| const fact = observeProviderKeyboard(params); | ||
| if (fact.kind !== 'unmeasurable' || !params.complete) return fact; |
There was a problem hiding this comment.
P2: This returns a derived fact for truncated trees whenever their partial key geometry passes the docking checks. Truncation can omit part of the keyboard, so the partial band can misclassify taps; return unmeasurable whenever complete is false.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/contracts/src/tap-keyboard-occlusion.ts, line 371:
<comment>This returns a `derived` fact for truncated trees whenever their partial key geometry passes the docking checks. Truncation can omit part of the keyboard, so the partial band can misclassify taps; return `unmeasurable` whenever `complete` is false.</comment>
<file context>
@@ -356,6 +358,31 @@ export function acquireProviderKeyboard(params: ProviderKeyboardInput): {
+ params: Readonly<{ nodes: readonly RawSnapshotNode[]; viewport: Rect | null; complete: boolean }>,
+): SnapshotKeyboardBandFact {
+ const fact = observeProviderKeyboard(params);
+ if (fact.kind !== 'unmeasurable' || !params.complete) return fact;
+ return params.nodes.some(isKeyboardTraceNode) ? fact : { kind: 'absent', treeBound: true };
+}
</file context>
| if (fact.kind !== 'unmeasurable' || !params.complete) return fact; | |
| if (!params.complete) { | |
| return { kind: 'unmeasurable', reason: 'keyboard-tree-truncated' }; | |
| } | |
| if (fact.kind !== 'unmeasurable') return fact; |
| 'agent-device': patch | ||
| --- | ||
|
|
||
| Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree, or a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar), stays `unmeasurable`. |
There was a problem hiding this comment.
P3: The changeset overstates truncated-tree behavior: a truncated capture can publish derived or tree-rejected, not only unmeasurable. Describe truncation as never proving absence, while preserving the fact established by its captured evidence.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .changeset/ios-ax-bridge-keyboard-fact.md, line 5:
<comment>The changeset overstates truncated-tree behavior: a truncated capture can publish `derived` or `tree-rejected`, not only `unmeasurable`. Describe truncation as never proving absence, while preserving the fact established by its captured evidence.</comment>
<file context>
@@ -0,0 +1,5 @@
+'agent-device': patch
+---
+
+Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree, or a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar), stays `unmeasurable`.
</file context>
| Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree, or a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar), stays `unmeasurable`. | |
| Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree never proves absence and preserves the fact established by its captured evidence; a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar) stays `unmeasurable`. |
| - The first read after `scroll`, `swipe`, or `gesture swipe` waits for two captures to agree. When that proves something about the gesture, `is`, `get`, `find`, `wait`, interactions that captured it, and `snapshot` add a warning, and all but `snapshot` report `postGestureOutcome` (`{ "kind", "gesture": { "action", "positionals" } }`) in `data` or `error.details`: | ||
| - `kind: "unsettled"`: the screen was still changing. A miss is not proof of absence; read again. `is absent` refuses with `observation: "unsettled"`, and `wait absent` keeps polling. | ||
| - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner does, a WebDriver (Appium) visibility probe does not. Otherwise only the content is compared. | ||
| - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window. Otherwise only the content is compared. |
There was a problem hiding this comment.
P3: The command documentation narrows the bridge’s absence rule too far. Mention Keyboard/Key nodes and iOS keyboard roles, because missing only the keyboard window does not prove dismissal.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At website/docs/docs/commands.md, line 610:
<comment>The command documentation narrows the bridge’s absence rule too far. Mention `Keyboard`/`Key` nodes and iOS keyboard roles, because missing only the keyboard window does not prove dismissal.</comment>
<file context>
@@ -607,7 +607,7 @@ agent-device is text 'id="greeting"' "Welcome back"
- The first read after `scroll`, `swipe`, or `gesture swipe` waits for two captures to agree. When that proves something about the gesture, `is`, `get`, `find`, `wait`, interactions that captured it, and `snapshot` add a warning, and all but `snapshot` report `postGestureOutcome` (`{ "kind", "gesture": { "action", "positionals" } }`) in `data` or `error.details`:
- `kind: "unsettled"`: the screen was still changing. A miss is not proof of absence; read again. `is absent` refuses with `observation: "unsettled"`, and `wait absent` keeps polling.
- - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner does, a WebDriver (Appium) visibility probe does not. Otherwise only the content is compared.
+ - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window. Otherwise only the content is compared.
## Replay
</file context>
| - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window. Otherwise only the content is compared. | |
| + - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window, `Keyboard` or `Key` node, or iOS keyboard role. Otherwise only the content is compared. | |
| (Based on your team's feedback about the keyboard evidence contract.) [149b3648-85e9-4aaf-a5cd-25cfdbd55004] |
|
At 05c8da9, the code looks good, and it follows what we agreed on #4207. The bridge proves keyboard absence only from a complete, untruncated tree; truncation never proves absence, and presence evidence is kept. Bridge taps now read the band the acquisition derived, like the runner and the providers do. Your base-versus-head numbers in the PR body cover the default simulator route. The tap-risk thread no longer applies. On base, a bridge capture had no fact and the band came from the same truncated, projected nodes. Now it comes from the raw truncated tree, which is a superset, and only absence needs a complete tree. Not blocking, take or leave: for a truncated tree with a docked keyboard, tap-keyboard-occlusion.ts:371 publishes The only failing check is the Vercel deployment authorization, which is not a code problem. No conflicts are known. |
Summary
Follow-up to #4207. On the default Simulator route (
simulator-ax-bridge), a drag that only dismisses the keyboard still waited for the 3.5 s distrust cap and returned a stale-accepted read: bridge captures published no keyboard fact.The bridge reads the app root like the runner, so the keyboard window is in its tree, and it marks every cut
truncated. A bridge capture now publishes a tree-bound fact (observeCompleteTreeKeyboard):derived);Keyboard/Keynode, or keyboard role:absent;unmeasurable.Taps do not change. 9 files. I can add an ADR 0004 amendment if wanted.
Validation
pnpm check:affected --base upstream/main --runat05c8da976: passed.7d42395ffvs head4d0176d5f(same code as05c8da976), iPhone 17 Pro simulator, iOS 26.4, 60 runs, all exit 0:no-effecton both builds, no false trust.AI assistance and human review
AI was used to prepare these changes. Human-reviewed by @kdy1.
🤖 Generated with Claude Code
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.