Skip to content

fix(server): find a finished sandbox's supervisor on any replica - #5667

Merged
pimlock merged 1 commit into
mainfrom
fix/4949-ssh-session-any-replica/pimlock
Nov 17, 2026
Merged

pimlock merged 1 commit into
mainfrom
fix/4949-ssh-session-any-replica/pimlock

Conversation

@pimlock

@pimlock pimlock commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

On a multi-replica gateway, attaching to a sandbox whose main process already finished could fail with sandbox is not ready. The post-exit check only looked for the supervisor session on the replica that received the request. It now also finds a supervisor connected to another replica, the same way relay opening already routes to the owning replica.

This is split out of #4950 so it can land first. #4950 keeps the rest of the post-exit access work.

Related Issue

Part of #4949. Fixes the case in #4949 (comment): sandbox create -- <short cmd> on the k3s-ha row, where the command finished before the CLI attached and CreateSshSession reached a replica without the supervisor session.

Changes

  • supervisor_session::supervisor_connected: true when this replica holds the session, or when a fresh owner record names a peer replica. On a single-replica gateway it doesn't read the owner record. An owner record that names this replica but has no local session is treated as stale.
  • CreateSshSession and ForwardTcp use it for Completed and Error sandboxes. Ready is unchanged, and so is every other phase.
  • Host gateway e2e: run the Python client as the sandbox's main process again, removing the Ready-sandbox exec workaround from test(tmachine): prepare runner environments for the e2e/rust migration #5277. The k3s-ha row now covers attaching to a finished main process across replicas.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

New unit tests cover supervisor_connected and the owner-record rule, and the existing CreateSshSession test runs against the new check. The host-gateway test compiles. I haven't run it live; the multi-replica case is exercised by the k3s-ha CI row.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

After a retained sandbox's main process exits, CreateSshSession and
ForwardTcp allow access only while the supervisor is connected. The check
looked only at the receiving replica's session map, so on a multi-replica
gateway a request that landed on another replica was refused with
"sandbox is not ready". This broke `sandbox create -- <cmd>` when the
command finished before the CLI attached.

Check the supervisor owner record too, the same way relay opening already
routes to the owning replica. Run the host-gateway client as the sandbox's
main process again, so the k3s-ha row covers the attach.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@pimlock pimlock added the test:e2e Requires end-to-end coverage label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for dac3026. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at dac3026. No correctness blockers found. The remaining validation gap is that the multi-replica E2E case was skipped in the current CI run.

@pimlock
pimlock added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit dd3febc Nov 17, 2026
203 checks passed
@pimlock
pimlock deleted the fix/4949-ssh-session-any-replica/pimlock branch November 17, 2026 23:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants