Skip to content

Proposal: Agent Substrate as a delegated isolation backend under RFC 12 #3362

Open

Description

@mayawang

Proposal: Agent Substrate as a delegated isolation backend

We're planning to implement RFC 12 for Agent Substrate, and Drew offered on #2048 to work through a proposal, so here is one. It's concrete about what we'd build and about the three places the contract would need to move first.

What Substrate is, briefly

An open-source system that runs agent workloads on Kubernetes by multiplexing many "actors" onto a smaller pool of ready worker pods. Agents are idle most of the time, so actors are checkpointed to storage when idle and restored on demand, which is where the density comes from. Isolation is gVisor. There's a control plane and a per-node agent; the node agent is what owns a sandbox's lifecycle.

The topology it lands as

Delegated, in the RFC's sense. Substrate is a separately deployed control plane, so the backend would be a thin in-process implementation of the interface that carries its own transport to the node agent, with delegation staying topology-private. That's the pattern the RFC already describes, in the alternatives section and again in non-goals, rather than anything new.

Three things in the contract

1. A transport surface for the delegated link. The alternatives section says a transport-bearing surface "is not precluded: it is versioned contract surface, added when a concrete delegated backend requires it." We'd be that backend, so this is us supplying the trigger rather than reopening the decision. What we'd want to agree is the shape: which operations cross the link, and what the versioning story is when the two halves are deployed separately and can skew.

2. Confirmation evidence as properties rather than mechanisms. SandboxConfirmEvidence::validate currently requires a specific set of seccomp user-notification facts plus landlock_abi >= 3. A gVisor boundary provides neither, because the sentry implements the syscall interception itself, so no backend on it can reach Ready regardless of the isolation it actually provides. The merged RFC doesn't ask for either mechanism, and its kernel-relationship table already contemplates a userspace application kernel, so I read this as a gap between the text and the crate. Written up properly in #3361 rather than here, with the measured probe output and a proposed shape.

3. A memory-preserving lifecycle verb, plus a capability bit. Today's lifecycle is cold: Stop and Start by design. Substrate suspends an actor with its memory intact and resumes it, which is the property the density model depends on, and there's no way to express it through the current verbs. GetCapabilities already negotiates gateway_manages_lifecycle, so negotiating lifecycle semantics per backend is an established pattern here, and a backend that doesn't implement the verb would simply not advertise it.

One thing I'd rather ask than assume: does BoundaryLoopbackConnector cover port forwarding into a delegated topology, or is that still open?

What we'd do

Build it, against a real deployment rather than a mock. We have Substrate running on GKE today with gVisor actors suspending and resuming with memory and filesystem state both intact, so the backend has something to sit on from day one.

We'd also be glad to contribute the delegated-topology scenarios to openshell-conformance as part of the work. A backend that's out of process is the awkward case for a suite, and it seems better for us to write those than to hand you a backend and leave the testing to someone else.

Happy to break any of the three out into its own issue if that's easier to review than one thread.

Activity

  1. mayawang commented on Sep 16, 2026

    @mayawang
    Author

    @drew this is the proposal half of what you asked for on #2048. #3361 is the narrow piece, on the confirmation evidence itself; this one is the delegated backend surface, the transport, and a lifecycle verb that preserves memory behind a capability bit.

    Flagging it because the only pointer to it was a cross-reference. Happy to split it into smaller issues if that reviews better.

  2. added
    retired:area:sandboxRetired; do not apply. See #5151 for item-specific migration to the current taxonomy.
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Sep 16, 2026
  3. github-actions commented on Oct 2, 2026

    @github-actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  4. danehans commented on Oct 7, 2026

    @danehans
    Contributor

    Hi all! I'm a +1 to see Substrate as a supported isolation backend. In general, I think it would be helpful to exercise the different OpenShell extension points.

  5. danehans commented on Oct 8, 2026

    @danehans
    Contributor

    This would let OpenShell agents free up compute during idle without losing state, while keeping its policy and credential controls in OpenShell. @drew with #3366 merged, will you be open to accepting a Substrate backend along these lines?

  6. iracic82 commented on Oct 22, 2026

    @iracic82

    I've been testing OpenShell with Agent Substrate on kind, mostly to see if we can avoid a supervisor pod per agent (#2385). One result changes the picture here a bit.

    The issue describes Substrate isolation as gVisor, and on gVisor I see what #3361 measured: runsc release-20261012.0 gives ENOSYS for Landlock and EINVAL for SECCOMP_FILTER_FLAG_NEW_LISTENER, so openshell-sandbox fails closed. But Substrate also has a microVM class (ateworker-microvm: Kata 4.1.0 guest kernel 6.18, Cloud Hypervisor), and there the stock runtime qualifies.

    Inside a Substrate microVM actor (substrate@ab4f3dbe), openshell-sandbox capability-probe-launch 1000 1000 from the 0.1.5 release returns "qualified": true with Landlock ABI 7. Same result after suspending and resuming the actor, 3 of 3 runs.

    Two things the actor needs, both on the Substrate side:

    So I think the microVM class could use the native backend as it is, and the delegated backend would only be needed for gVisor actors. Maybe that's a smaller first step.

    I also ran the shared supervisor from #4702 on Kubernetes (results on that PR). Per sandbox it goes from 2 pods and 16.4 MiB of supervisor memory to 1 pod and about 5.4 MiB. Wiring that to Substrate actors needs a driver path, which is what this issue is about.

    @drew is a microVM-first path something you'd consider here? If yes, I'm happy to work on it.

  7. JimBugwadia commented on Nov 17, 2026

    @JimBugwadia

    Hi @mayawang @iracic82 - now that agent-substrate is 1.0 what is the next step for this initiative? Thanks!

  8. iracic82 commented on Nov 17, 2026

    @iracic82

    Yo Jim, good timing. I re-ran everything on the 1.0 release before answering, so here's where it actually stands.

    The microVM path works on 1.0. Substrate v1.0.0, microVM class (Kata 4.3.1, guest kernel 6.18.56), and the stock openshell-sandbox binary from the v0.1.8 release: capability-probe-launch comes back "qualified": true with Landlock ABI 7 and seccomp notification working. 3 of 3 runs, on a fresh actor and again after suspend and resume. The only thing the template sets is capabilities: {drop: [ALL], add: [SETPCAP, SETUID, SETGID]}. No sysctl hack anymore, since agent-substrate/substrate#1904 shipped in 1.0, and a counter actor keeps its in-memory state across the suspend.

    Two 1.0 changes to know about. Containers now run as the image's USER (agent-substrate/substrate#2392), and the OpenShell launcher has to start as root: an image with USER 1000 fails closed with "capability-free launch must start as root". Easy to catch when the template is created. And capabilities.add is now an allowlist (agent-substrate/substrate#3580); the three caps above are all on it.

    OpenShell moved a lot too since Maya's proposal:

    So the contract pieces are mostly there. What's missing is the Substrate driver itself.

    To be upfront about scope: what I tested is OpenShell's own runtime qualification inside a Substrate actor. I haven't run a full sandbox from the gateway into an actor yet, because that's exactly the driver's job.

    What I'd like to do next:

    1. An external Substrate compute driver, microVM class first, that maps SuspendSandbox/ResumeSandbox onto Substrate's suspend and resume. That's where the density win comes from.
    2. Maya's gVisor delegated backend after that, on top of the isolation backend service. @mayawang, still planning on it?

    @drew two questions before I start. Are you good with this living out of tree behind the driver socket, or would you rather have it in-tree? And for this placement the driver would own the outer fence: what evidence would you want for default-deny egress and fail-closed controller loss? Substrate has its own egress gateway, but I haven't tested it against your fence guarantees, so I'd rather agree on that before writing code.

  9. JimBugwadia commented on Nov 17, 2026

    @JimBugwadia

    Thanks @iracic82 - this is great! Is there an RFC and / or a prototype available to try?

    If this is out-of-tree, what would the user install experience be? Separately install substrate and the driver, after installing the OpenShell chart?

    On the egress gateway and rules, substrate has an internal policy and there was a proposal on using OpenFGA and allowing external engines. OpenShell also has its own YAML format, and is moving towards Cedar. For an OpenShell user, it would be best to stay with OpenShell defined APIs. How do you see these aligning?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    retired:area:sandboxRetired; do not apply. See #5151 for item-specific migration to the current taxonomy.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions