Skip to content

bug(supervisor): exec fails in bursts when a shared supervisor is killed or restarted #5659

Open

Description

@FrostGod

User Story

I ran a mini scale test of OpenShell on Kubernetes with a shared supervisor pool and kept openshell sandbox exec traffic running against the sandboxes. When a supervisor was killed or restarted, hundreds of execs failed even though the sandboxes recovered. I need supervisor restarts and upgrades to be invisible, or close to it, for callers that run commands in sandboxes.

Problem Statement

When a shared supervisor goes away and its sandboxes move to another supervisor, execs fail in a burst:

  • SIGKILL of one supervisor: about 220 exec failures within about 10 seconds, then full recovery.
  • Rolling restart of the supervisors: about 415 failures over about 20 seconds. Some sandboxes moved twice, and one sandbox ended up permanently Stopped.

The errors:

  • Most common: sandbox is not ready (phase: Provisioning)
  • Second: relay open timed out, from execs that were already running on the supervisor that died

Why This Happens

  1. The sandbox is marked not ready during the handoff. When a supervisor's session ends, the gateway moves the sandbox from Ready back to Provisioning (finish_supervisor_session). It stays there until the new supervisor takes the sandbox and reconnects. There is a short grace period that keeps the sandbox Ready, but it only applies when a gateway shuts down and redirects its supervisors. It doesn't apply when a supervisor dies or restarts.
  2. Exec gives up right away. openshell sandbox exec checks the phase first and fails at once if it isn't Ready, with no retry (run.rs). A handoff that takes a few seconds therefore turns into a burst of failures.
  3. Rolling restarts move sandboxes twice. A new home for a sandbox is picked from all live supervisors (supervisor_pool.rs). The pick doesn't prefer supervisors that have already been restarted, so a sandbox can land on an old supervisor that is restarted next and has to move again.
  4. One sandbox ended up Stopped. I haven't found the cause yet. It may be related to bug(supervisor): gate rollout readiness on gateway attachment #4960 (supervisors reporting ready before they attach to the gateway).

Impact / Why This Matters

  • Every supervisor crash, node drain, or upgrade causes a burst of failed commands for every sandbox on that supervisor.
  • Agents and scripts that call exec see hard failures, not a short delay, unless they write their own retry logic.
  • Moving twice during rollouts doubles the disruption, and a sandbox left Stopped needs a manual restart.

Acceptance Criteria

  • During a single supervisor takeover, openshell sandbox exec waits for the sandbox to come back for a bounded time instead of failing immediately.
  • A rolling restart of the supervisor pool moves each sandbox at most once.
  • No sandbox ends up Stopped after a rolling restart.
  • A regression test runs exec traffic through a supervisor SIGKILL and a rolling restart, and checks the failure count and recovery.

Reproduction Steps

  1. Deploy OpenShell on Kubernetes with a shared supervisor pool of at least 2 supervisors.
  2. Create several sandboxes and run openshell sandbox exec <name> -- true in a loop against each one.
  3. Kill one supervisor pod (kubectl delete pod --grace-period=0 --force <supervisor-pod>), or start a rolling restart of the supervisors (kubectl rollout restart deployment/<supervisor-deployment>).
  4. Watch the exec failures (sandbox is not ready (phase: Provisioning), relay open timed out) and the sandbox phases (openshell sandbox list).

Environment

  • OpenShell: mini scale test build; code paths confirmed on main at 624acee
  • Deployment: Kubernetes (Helm), multi-replica gateway, shared supervisor pool, PostgreSQL

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions