Skip to content

fix(ios): give simulator AX bridge captures a tree-bound keyboard fact - #4217

Merged
thymikee merged 4 commits into
callstack:mainfrom
kdy1:fix/ios-ax-bridge-keyboard-fact
Nov 4, 2026
Merged

thymikee merged 4 commits into
callstack:mainfrom
kdy1:fix/ios-ax-bridge-keyboard-fact

Conversation

@kdy1

@kdy1 kdy1 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #4207. On the default Simulator route (simulator-ax-bridge), a drag that only dismisses the keyboard still waited for the 3.5 s distrust cap and returned a stale-accepted read: bridge captures published no keyboard fact.

The bridge reads the app root like the runner, so the keyboard window is in its tree, and it marks every cut truncated. A bridge capture now publishes a tree-bound fact (observeCompleteTreeKeyboard):

  • a docked keyboard: the docked-tree band (derived);
  • a complete tree with no keyboard window, Keyboard/Key node, or keyboard role: absent;
  • a truncated tree, or a keyboard window without keys (placeholder, assistant bar only): unmeasurable.

Taps do not change. 9 files. I can add an ADR 0004 amendment if wanted.

Validation

  • pnpm check:affected --base upstream/main --run at 05c8da976: passed.
  • Live, base 7d42395ff vs head 4d0176d5f (same code as 05c8da976), iPhone 17 Pro simulator, iOS 26.4, 60 runs, all exit 0:
    • Default route, keyboard dismiss ×15: stale-accept 3701 ms → trust 770 ms (median read), 13 → 3 captures.
    • Default route, no-keyboard control ×10: no-effect on both builds, no false trust.
    • Runner route ×5: unchanged (860 / 877 ms).
  • Not covered live: truncated and window-without-keys branches (unit tests only), physical devices, iPad, other keyboards and runtimes.

AI assistance and human review

AI was used to prepare these changes. Human-reviewed by @kdy1.

  • Harness: T3 Code (Claude Code)
  • Model: claude-opus-5-5

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff

kdy1 and others added 4 commits November 4, 2026 21:51
The default iOS Simulator route (`simulator-ax-bridge`) published no
keyboard fact. Post-gesture stabilization counts a keyboard change only
between two tree-bound facts, so a drag that only dismissed the keyboard
still waited for the distrust cap (about 3.7 s) on that route.

The bridge reads the app's root with the same AX snapshot API as the
runner, and the system keyboard window is one of the app's own windows in
that tree. The adapter now publishes `observeCompleteTreeKeyboard`: the
docked-tree band when the tree holds a docked keyboard, and `absent` with
`treeBound` only when the tree is complete and holds no keyboard trace (no
`Keyboard`/`Key` node and no iOS keyboard role, which includes the remote
keyboard placeholder and the input assistant). A truncated tree, or one
that holds only a trace, publishes no absence.

Tests pin the adapter's fact for docked, complete, truncated,
placeholder-only, and assistant-only trees; add complete-tree rows to the
tap keyboard occlusion table; and drive the stabilization loop with
bridge-shaped `derived -> absent` (trusts) and unchanged
`absent -> absent` / `derived -> derived` pairs (fall back to content).
The bridge gets its own changeset, and the `no-effect` docs and ADR 0004
now name the bridge as a follow-up to the tree-bound keyboard change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A complete Simulator AX bridge tree that held a `UIRemoteKeyboardWindow`
with no keys published `absent` with `treeBound`: the window class was not
a keyboard trace. A keyboard window that is present but exposes no keys,
for example a hardware keyboard that shows only the assistant bar, is a
partial reading, so it must stay `unmeasurable`.

The keyboard window now counts as a keyboard trace. Only a complete tree
with no keyboard window, no `Keyboard` or `Key` node, and no keyboard role
publishes absence.

Adapter tests and golden rows pin a keyboard window without keys, with only
the assistant bar, and with only the remote keyboard placeholder. The
changeset, the `no-effect` docs, and ADR 0004 name the keyboard window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rebase onto the surface-host check took adapter.test.ts past the 1000-line cap. The guest
trees now live beside it, and the setup reads SNAPSHOT_BRIDGE_SOURCE_FILENAMES, so it also
writes the surface-host sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bridge fact contradicts no statement in ADR 0004, and the commands docs, the changeset and
the code comments already record the behavior. The provider-producer sentence that callstack#4207 made
stale is outside this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 4, 2026

Copy link
Copy Markdown

@kdy1 is attempting to deploy a commit to the Callstack Team on Vercel.

A member of the Team first needs to authorize it.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 9 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/contracts/src/tap-keyboard-occlusion.ts">

<violation number="1" location="packages/contracts/src/tap-keyboard-occlusion.ts:371">
P2: This returns a `derived` fact for truncated trees whenever their partial key geometry passes the docking checks. Truncation can omit part of the keyboard, so the partial band can misclassify taps; return `unmeasurable` whenever `complete` is false.</violation>
</file>

<file name=".changeset/ios-ax-bridge-keyboard-fact.md">

<violation number="1" location=".changeset/ios-ax-bridge-keyboard-fact.md:5">
P3: The changeset overstates truncated-tree behavior: a truncated capture can publish `derived` or `tree-rejected`, not only `unmeasurable`. Describe truncation as never proving absence, while preserving the fact established by its captured evidence.</violation>
</file>

<file name="website/docs/docs/commands.md">

<violation number="1" location="website/docs/docs/commands.md:610">
P3: The command documentation narrows the bridge’s absence rule too far. Mention `Keyboard`/`Key` nodes and iOS keyboard roles, because missing only the keyboard window does not prove dismissal.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

params: Readonly<{ nodes: readonly RawSnapshotNode[]; viewport: Rect | null; complete: boolean }>,
): SnapshotKeyboardBandFact {
const fact = observeProviderKeyboard(params);
if (fact.kind !== 'unmeasurable' || !params.complete) return fact;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This returns a derived fact for truncated trees whenever their partial key geometry passes the docking checks. Truncation can omit part of the keyboard, so the partial band can misclassify taps; return unmeasurable whenever complete is false.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/contracts/src/tap-keyboard-occlusion.ts, line 371:

<comment>This returns a `derived` fact for truncated trees whenever their partial key geometry passes the docking checks. Truncation can omit part of the keyboard, so the partial band can misclassify taps; return `unmeasurable` whenever `complete` is false.</comment>

<file context>
@@ -356,6 +358,31 @@ export function acquireProviderKeyboard(params: ProviderKeyboardInput): {
+  params: Readonly<{ nodes: readonly RawSnapshotNode[]; viewport: Rect | null; complete: boolean }>,
+): SnapshotKeyboardBandFact {
+  const fact = observeProviderKeyboard(params);
+  if (fact.kind !== 'unmeasurable' || !params.complete) return fact;
+  return params.nodes.some(isKeyboardTraceNode) ? fact : { kind: 'absent', treeBound: true };
+}
</file context>
Suggested change
if (fact.kind !== 'unmeasurable' || !params.complete) return fact;
if (!params.complete) {
return { kind: 'unmeasurable', reason: 'keyboard-tree-truncated' };
}
if (fact.kind !== 'unmeasurable') return fact;

'agent-device': patch
---

Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree, or a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar), stays `unmeasurable`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The changeset overstates truncated-tree behavior: a truncated capture can publish derived or tree-rejected, not only unmeasurable. Describe truncation as never proving absence, while preserving the fact established by its captured evidence.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .changeset/ios-ax-bridge-keyboard-fact.md, line 5:

<comment>The changeset overstates truncated-tree behavior: a truncated capture can publish `derived` or `tree-rejected`, not only `unmeasurable`. Describe truncation as never proving absence, while preserving the fact established by its captured evidence.</comment>

<file context>
@@ -0,0 +1,5 @@
+'agent-device': patch
+---
+
+Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree, or a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar), stays `unmeasurable`.
</file context>
Suggested change
Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree, or a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar), stays `unmeasurable`.
Fix the first read after a drag that only shows or hides the soft keyboard waiting about 3.5 s on the iOS Simulator AX bridge, the default Simulator route. A bridge capture now publishes a keyboard fact read from its own tree: the docked-tree band when the tree holds a docked keyboard, and `absent` only when the tree is complete and holds no keyboard window, `Keyboard` or `Key` node, or keyboard role. A truncated tree never proves absence and preserves the fact established by its captured evidence; a keyboard window that exposes no keys (for example, a hardware keyboard that shows only the assistant bar) stays `unmeasurable`.

- The first read after `scroll`, `swipe`, or `gesture swipe` waits for two captures to agree. When that proves something about the gesture, `is`, `get`, `find`, `wait`, interactions that captured it, and `snapshot` add a warning, and all but `snapshot` report `postGestureOutcome` (`{ "kind", "gesture": { "action", "positionals" } }`) in `data` or `error.details`:
- `kind: "unsettled"`: the screen was still changing. A miss is not proof of absence; read again. `is absent` refuses with `observation: "unsettled"`, and `wait absent` keeps polling.
- `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner does, a WebDriver (Appium) visibility probe does not. Otherwise only the content is compared.
- `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window. Otherwise only the content is compared.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The command documentation narrows the bridge’s absence rule too far. Mention Keyboard/Key nodes and iOS keyboard roles, because missing only the keyboard window does not prove dismissal.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At website/docs/docs/commands.md, line 610:

<comment>The command documentation narrows the bridge’s absence rule too far. Mention `Keyboard`/`Key` nodes and iOS keyboard roles, because missing only the keyboard window does not prove dismissal.</comment>

<file context>
@@ -607,7 +607,7 @@ agent-device is text 'id="greeting"' "Welcome back"
 - The first read after `scroll`, `swipe`, or `gesture swipe` waits for two captures to agree. When that proves something about the gesture, `is`, `get`, `find`, `wait`, interactions that captured it, and `snapshot` add a warning, and all but `snapshot` report `postGestureOutcome` (`{ "kind", "gesture": { "action", "positionals" } }`) in `data` or `error.details`:
   - `kind: "unsettled"`: the screen was still changing. A miss is not proof of absence; read again. `is absent` refuses with `observation: "unsettled"`, and `wait absent` keeps polling.
-  - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner does, a WebDriver (Appium) visibility probe does not. Otherwise only the content is compared.
+  - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window. Otherwise only the content is compared.
 
 ## Replay
</file context>
Suggested change
- `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window. Otherwise only the content is compared.
+ - `kind: "no-effect"`: nothing changed. The list may be at its edge or ignore synthesized scrolls; a raw `swipe x1 y1 x2 y2` inside it moves such lists. The soft keyboard appearing or going away counts as a change only when the pre-gesture baseline and a later quiet capture both read the keyboard from their own tree and disagree: the iOS XCTest runner and the iOS Simulator AX bridge (the default Simulator route) do, a WebDriver (Appium) visibility probe does not. The AX bridge reads the keyboard as gone only from a complete tree that holds no keyboard window, `Keyboard` or `Key` node, or iOS keyboard role. Otherwise only the content is compared.
(Based on your team's feedback about the keyboard evidence contract.) [149b3648-85e9-4aaf-a5cd-25cfdbd55004]

@thymikee

thymikee commented Nov 4, 2026

Copy link
Copy Markdown
Member

At 05c8da9, the code looks good, and it follows what we agreed on #4207. The bridge proves keyboard absence only from a complete, untruncated tree; truncation never proves absence, and presence evidence is kept. Bridge taps now read the band the acquisition derived, like the runner and the providers do. Your base-versus-head numbers in the PR body cover the default simulator route.

The tap-risk thread no longer applies. On base, a bridge capture had no fact and the band came from the same truncated, projected nodes. Now it comes from the raw truncated tree, which is a superset, and only absence needs a complete tree.

Not blocking, take or leave: for a truncated tree with a docked keyboard, tap-keyboard-occlusion.ts:371 publishes derived or tree-rejected, so "unmeasurable" and "Taps do not change" in the PR body are not exact. A contract fixture row for that case would pin it. Two lower-priority wording threads on the changeset and the commands.md note still apply.

The only failing check is the Vercel deployment authorization, which is not a code problem. No conflicts are known.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 4, 2026
@thymikee
thymikee merged commit f5b84ec into callstack:main Nov 4, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants