Skip to content

fix(apple): recover runner snapshot bridge after a wedge or restart - #4256

Merged
thymikee merged 5 commits into
mainfrom
fix/runner-bridge-wedge-and-restart
Nov 6, 2026
Merged

thymikee merged 5 commits into
mainfrom
fix/runner-bridge-wedge-and-restart

Conversation

@thymikee

@thymikee thymikee commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Summary

Retire a runner's snapshot bridge when an abandoned reader remains blocked for five seconds. Each runner session owns its circuit and target cache, so a replacement retries the bridge and late failures cannot retire the replacement's generations. Session-scoped refusals retain their scope in warnings and diagnostics. Fixes #4241.

15 files changed across the runner bridge, routing, tests, refusal fixture, and ADR.

Validation

Base: b42c7c265f0765045b1ce5c53dd426512e11108d. Head: e25fef03950b54543ca29a71fbfaede8f4586cd4.

  • Original regression tests failed on base and passed at 4e78ccc; the updated focused suites pass at this head (84 tests).
  • Xcode 26.2 iOS Simulator native test passed: abandoned → in-flight before five seconds → wedged at five seconds → admitted after release. The reader was held on a semaphore and the clock was controlled.
  • pnpm check:affected --run --head e25fef03950b54543ca29a71fbfaede8f4586cd4 passed: run-pvhLTv, exactCommit=true, checkoutChanged=false. Coverage and native/device lanes remain GitHub-owned or skipped; provider integration was deduplicated against related tests.

Live evidence: on Limrun iOS at e25fef0, a real bridge-read-abandoned refusal (triggered by a 1 ms budget) retired the generation. After the runner was replaced through stopIosRunnerSession, the route logged the session change and served simulator-ax-bridge for the same app generation with no warning. The PR comment has the details. No data shows finite remote reads crossing 5 s: no Limrun run has returned an abandoned, deadline, or wedged refusal without fault injection.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.14 MB 6.15 MB +3.1 kB
Package (unpacked) 6.14 MB 6.14 MB +3.1 kB
Package (download) 1.86 MB 1.86 MB +664 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 21.2 ms 25.3 ms +4.0 ms
CLI --help 44.5 ms 49.7 ms +5.2 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Apex Sentinel

Verdict: approve

Since the last review: 0 new · 0 still open · 2 no longer raised

No findings.

2 findings below minor not shown (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (0a1d37cb5359).

Reply under an inline comment with @apex-sentinel-staging note ... or @apex-sentinel-staging pivot ... to re-check that finding.
Comment @apex-sentinel-staging review on this conversation to run a full review.

Earlier reviews

Review of 6dcfd3fe · 06.11.2026 09:55 UTC

Apex Sentinel

Verdict: approve

Since the last review: 2 new · 0 still open · 2 no longer raised

2 finding(s): 2 minor.

🟡 Minor (2)

Where Finding
packages/platform-apple/src/snapshot-route.ts:208 🆕 sessionDisabled disables the bridge route forever when no probe names a session
packages/platform-apple/src/snapshot-route.ts:208 🆕 Only a channel that passes sessionChanged can ever clear sessionDisabled, so a local-route wedge is permanent

3 findings below minor not shown (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (0a1d37cb5359).

Review of e25fef03 · 06.11.2026 07:46 UTC

Apex Sentinel

Verdict: request_changes

Since the last review: 2 new · 0 still open · 1 no longer raised

2 finding(s): 1 major, 1 minor.

Required fixes

  • A bridge-reader-wedged refusal on the describe path never retires the runner session

🟠 Major (1)

Where Finding
packages/platform-apple/src/runner-snapshot-bridge.test.ts:514 🆕 A bridge-reader-wedged refusal on the describe path never retires the runner session

🟡 Minor (1)

Where Finding
packages/platform-apple/src/snapshot-route.ts:202 🆕 circuitForProbe silently drops wedge evidence when the probe names a third runner session

Reviewed with this repository's own Sentinel configuration (90c959518e46).

Review of 4e78ccc1 · 06.11.2026 05:21 UTC

Apex Sentinel

Verdict: approve

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
packages/platform-apple/src/snapshot-route.ts:248 Post-wedge captures warn with the app-generation sentence, not the runner-session one

1 finding below minor not shown (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (b42c7c265f07).

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on 4e78ccc. The verdict and the full list are in the summary comment.

@thymikee

thymikee commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

The change reads sound at 4e78ccc, but the new runner recovery path has no live or native run, so I can't confirm it works. All 17 checks pass at this commit, but hosted CI does not run the native wedge path or a remote runner restart. Before merge, please add the evidence below and fix the two open warning and write-guard threads.

The central transition runs nowhere: a held read goes abandoned, then bridge-read-in-flight before 5 s, then bridge-reader-wedged at or after 5 s, then is admitted again after release (RunnerSnapshotBridge.m#L216). The TS tests assign the refusal code to a fake runner, and the native suite has no wedge case. A wrong uptime comparison, a missed sequence check or a never-cleared abandon time would ship unnoticed. The first sign would be a production session that pays a refused round trip on every capture, or that retires the bridge on a healthy runner. Please provide both: (a) a native test in RunnerTests+SnapshotBridgeTests.swift that drives RunnerSnapshotBridgeRead through a test-only seam with the read block held on a semaphore. With a 100 ms budget it should show the first call returns bridge-read-abandoned, a call before 5 s returns bridge-read-in-flight, a call at or after 5 s returns bridge-reader-wedged, and after release the next call is admitted and returns ok. (b) A live run at this commit on a remote (Limrun) iOS session, which uses createAppleRunnerRuntime. Open an app and take a snapshot served by simulator-ax-bridge. Get its generation retired (for example with a bridge-read-abandoned refusal) or otherwise show the circuit open. Replace the runner so readRunnerSessionLiveness reports a new session id, then take the next snapshot. The proof is that snapshot showing producer simulator-ax-bridge with no circuit-disabled warning, and the daemon log showing the session id change.

Could the circuit be one object per runner session instead? The route core would hold { sessionId, disabledGenerations, latestGeneration, sessionDisabled }, the probe or channel would return the session id it read through, and the route would swap in a fresh circuit when that id differs. Each capture would take the current circuit at its start and write only to it, so a late write from an old session lands in a discarded object. That would delete resetRunnerSession from SnapshotBridgeRoute, both epoch counters, the throwaway new Set() at snapshot-route.ts#L335, and the forward reference in runner-runtime.ts. It would also cover a wedge seen before any session id was known. What would have to change first is that SystemSurfaceForegroundProbe or SnapshotBridgeChannel must carry the session id it read; today readySession computes it and createSystemSurfaceForegroundProbe drops it.

Not blocking, take or leave: a runner-session disablement should belong to the session it was seen under, so runner-runtime.ts#L78 should clear it when the probe reads a different id, even if no id was recorded at the wedge (I could not confirm the remote path reaches this state); the permanent retirement at snapshot-route.ts#L245 could be confirmed in the ADR or bounded so one capture per interval retries the bridge; createAppleSnapshotRoute (the local AX route, #L86) exposes a reset nothing calls, so keep it off the shared route type; and the inline if at snapshot-route-failure-policy.ts#L79 could be a RUNNER_SESSION_SCOPED set beside CAPTURE_SCOPED_REFUSALS that mirrors the fixture.

On the open inline threads: the warning-scope thread (#4256 (comment)) and the unguarded presentation-failure write (#4256 (comment)) still apply. Five lower-priority threads still apply too. The launch-observation thread (#4256 (comment)) does not apply: that probe exists only in the local AX route, whose reader never emits bridge-reader-wedged, so you can resolve it.

One open question: do you know how often a slow but finite read on a remote Simulator crosses the 5 s bound?

@thymikee

thymikee commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

Native evidence for e25fef03950b54543ca29a71fbfaede8f4586cd4: Xcode 26.2 built the iOS runner with unit tests, and xcodebuild test-without-building passed RunnerTests/testAbandonedReadBecomesWedgedUntilItReturns on an iOS 26.2 Simulator (1 test, 0 failures). With a 100 ms read budget and held semaphore, the test checks bridge-read-abandoned, bridge-read-in-flight at 4.999 s, bridge-reader-wedged at 5 s, and an admitted successful read after release. The simulator was shut down afterward.

The exact-head affected gate passed (run-pvhLTv, exactCommit=true, checkoutChanged=false). The required Limrun circuit-open/replacement evidence is still missing: the available device host inventory exposes only the local host, so I could not run the remote simulator through the required device controls. I have no data on how often a slow but finite remote read crosses five seconds.

@thymikee

thymikee commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

Live Limrun iOS run at e25fef0 (createAppleRunnerRuntime, a runner built from this branch with Xcode 26.2, --debug, secrets redacted, instance terminated afterwards).

No real trigger retires a remote generation on demand, so the run used two local, uncommitted changes to the built JS, which were removed afterwards with pnpm build:

  1. The first bridge describe request sends maxDurationMs: 1. The runner itself then abandons the read and returns a real bridge-read-abandoned refusal.
  2. A flag file makes the next capture's foreground probe call the production stopIosRunnerSession(deviceId). The probe's existing uptime path then starts a new runner.
# Command Diagnostics Result
1 snapshot -i runner_session_changed (none → a4414230…), fallback reason bridge-read-abandoned, generation 55555:1793951106.98 XCTest; warning "…(bridge-read-abandoned); used XCTest for this app generation."
2 snapshot -i fallback reason circuit-disabled, same generation XCTest; warning "…(circuit-disabled)…"
– runner stopped through stopIosRunnerSession
3 snapshot -i ios_snapshot_route_runner_session_changed {previousSessionId: a4414230…, sessionId: 94bd40d5…}, then ios.snapshot-source.acquire/present producer simulator-ax-bridge, same generation 55555:1793951106.98 simulator-ax-bridge, no warning
4 snapshot -i acquire/present simulator-ax-bridge, same session and generation simulator-ax-bridge, no warning

The app generation did not change, so the new runner session alone reopened the bridge. Before this PR, the generation stayed retired.

Two side observations:

  • open <other app> --relaunch did not replace the Limrun runner in a debug run: session f40a98bd… stayed the same throughout. The circuit correctly stayed open in that case. That is why the run above stops the runner directly.
  • Your question on slow reads: I have no data on finite remote reads that cross 5 s. In this run, and in every Limrun run for feat(apple): serve the Simulator AX bridge from the XCTest runner #4182 (including Bluesky feeds and deep trees), the bridge never returned bridge-read-abandoned, bridge-request-deadline, or bridge-reader-wedged. The only bridge refusal seen without fault injection was snapshot-tree-changed, which is capture-scoped.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 new findings on e25fef0. The verdict and the full list are in the summary comment.

Both branches repeated the same guard prefix. Read the probed session id once,
compare it with the starting and current circuits, and document that a capture
writes through the circuit object itself, never a copy.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 new findings on 6dcfd3f. The verdict and the full list are in the summary comment.

…cuit names it

Only a probe that names a different runner session replaces a circuit, so a
runner-session refusal on a circuit with no session id disabled the bridge for
good. Such a refusal now speaks for its capture only.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

…o retire

A runner-session refusal on a circuit with no session id now reports a
capture-scoped bridge-read-busy disposition, so its warning no longer claims a
runner-session retirement that did not happen.
@thymikee

thymikee commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

This PR is ready at b9a16c4. The findings from the earlier review (#4256 (comment)) are fixed. CI shows 17 checks, none failing, and there are no conflicts. Nothing else needs to happen before merge.

Your live Limrun run was at e25fef0; the change since then is behavior-equivalent on that route, so it still covers it. A real bridge-reader-wedged refusal is covered by the native unit test and the TS fakes, not by a live run.

Not blocking, take or leave: (1) SystemSurfaceForegroundProbe still returns either a bare foreground or { foreground, sessionId }, so the route narrows it in three places (:215, :268, :466); if the probe always returned { foreground, sessionId? }, those narrowings could go. (2) The session-less fakes in runner-snapshot-bridge.test.ts:168 and :653 return 'background', but the real probe returns 'unknown' when no runner session is ready. (3) The native test in RunnerTests+SnapshotBridgeTests.swift:69 would stay green if both RunnerSnapshotBridgeAbandonedAt = 0 clears were removed; holding one more read after the admitted read, then calling at a time far past the old stamp and asserting bridge-read-in-flight, would pin that.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 6, 2026
@thymikee
thymikee merged commit 08f7234 into main Nov 6, 2026
17 checks passed
@thymikee
thymikee deleted the fix/runner-bridge-wedge-and-restart branch November 6, 2026 10:52
thymikee added a commit that referenced this pull request Nov 6, 2026
- SystemSurfaceForegroundProbe always returns { foreground, sessionId? }, so the
  snapshot route drops its three narrowings.
- Test fakes without a runner session answer 'unknown', as the real probe does.
- The native wedge test holds a read after the admitted one and checks it at a
  time far past the old abandon stamp, so it fails if the stamp is not cleared.
- The lease handler uses errorMessage() for the teardown diagnostic.

Follow-up to #4256 and #4257.
thymikee added a commit that referenced this pull request Nov 6, 2026
…4272)

* fix(apple): keep runner bridge test hooks out of the shipped package

#4256 put RunnerSnapshotBridgeReadForTest and RunnerSnapshotBridgeDrainForTest
under an #if AGENT_DEVICE_RUNNER_UNIT_TESTS block in the shipped
RunnerSnapshotBridge.m. The packager strips that block only from Swift, so the
macro survived packaging and check-package failed the Release Build job.

Move the hooks to UnitTests/RunnerSnapshotBridgeTesting.m, which the package
omits, and reach the read and the reader queue through a private
UnitTests/RunnerSnapshotBridge+Internal.h.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(apple): mark the runner bridge read's nil-able result nullable

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runner-served bridge: recover from a wedged reader and reset the circuit on runner restart

1 participant