Repository navigation
Open
Description
User Story
I ran a mini scale test of OpenShell on Kubernetes with a shared supervisor pool and kept openshell sandbox exec traffic running against the sandboxes. When a supervisor was killed or restarted, hundreds of execs failed even though the sandboxes recovered. I need supervisor restarts and upgrades to be invisible, or close to it, for callers that run commands in sandboxes.
Problem Statement
When a shared supervisor goes away and its sandboxes move to another supervisor, execs fail in a burst:
- SIGKILL of one supervisor: about 220 exec failures within about 10 seconds, then full recovery.
- Rolling restart of the supervisors: about 415 failures over about 20 seconds. Some sandboxes moved twice, and one sandbox ended up permanently
Stopped.
The errors:
- Most common:
sandbox is not ready (phase: Provisioning) - Second:
relay open timed out, from execs that were already running on the supervisor that died
Why This Happens
- The sandbox is marked not ready during the handoff. When a supervisor's session ends, the gateway moves the sandbox from
Readyback toProvisioning(finish_supervisor_session). It stays there until the new supervisor takes the sandbox and reconnects. There is a short grace period that keeps the sandboxReady, but it only applies when a gateway shuts down and redirects its supervisors. It doesn't apply when a supervisor dies or restarts. - Exec gives up right away.
openshell sandbox execchecks the phase first and fails at once if it isn'tReady, with no retry (run.rs). A handoff that takes a few seconds therefore turns into a burst of failures. - Rolling restarts move sandboxes twice. A new home for a sandbox is picked from all live supervisors (
supervisor_pool.rs). The pick doesn't prefer supervisors that have already been restarted, so a sandbox can land on an old supervisor that is restarted next and has to move again. - One sandbox ended up
Stopped. I haven't found the cause yet. It may be related to bug(supervisor): gate rollout readiness on gateway attachment #4960 (supervisors reporting ready before they attach to the gateway).
Impact / Why This Matters
- Every supervisor crash, node drain, or upgrade causes a burst of failed commands for every sandbox on that supervisor.
- Agents and scripts that call
execsee hard failures, not a short delay, unless they write their own retry logic. - Moving twice during rollouts doubles the disruption, and a sandbox left
Stoppedneeds a manual restart.
Acceptance Criteria
- During a single supervisor takeover,
openshell sandbox execwaits for the sandbox to come back for a bounded time instead of failing immediately. - A rolling restart of the supervisor pool moves each sandbox at most once.
- No sandbox ends up
Stoppedafter a rolling restart. - A regression test runs exec traffic through a supervisor SIGKILL and a rolling restart, and checks the failure count and recovery.
Reproduction Steps
- Deploy OpenShell on Kubernetes with a shared supervisor pool of at least 2 supervisors.
- Create several sandboxes and run
openshell sandbox exec <name> -- truein a loop against each one. - Kill one supervisor pod (
kubectl delete pod --grace-period=0 --force <supervisor-pod>), or start a rolling restart of the supervisors (kubectl rollout restart deployment/<supervisor-deployment>). - Watch the exec failures (
sandbox is not ready (phase: Provisioning),relay open timed out) and the sandbox phases (openshell sandbox list).
Environment
- OpenShell: mini scale test build; code paths confirmed on
mainat 624acee - Deployment: Kubernetes (Helm), multi-replica gateway, shared supervisor pool, PostgreSQL
Metadata
Metadata
Assignees
Labels
No labels
Activity