Repository navigation
Description
Proposal: Agent Substrate as a delegated isolation backend
We're planning to implement RFC 12 for Agent Substrate, and Drew offered on #2048 to work through a proposal, so here is one. It's concrete about what we'd build and about the three places the contract would need to move first.
What Substrate is, briefly
An open-source system that runs agent workloads on Kubernetes by multiplexing many "actors" onto a smaller pool of ready worker pods. Agents are idle most of the time, so actors are checkpointed to storage when idle and restored on demand, which is where the density comes from. Isolation is gVisor. There's a control plane and a per-node agent; the node agent is what owns a sandbox's lifecycle.
The topology it lands as
Delegated, in the RFC's sense. Substrate is a separately deployed control plane, so the backend would be a thin in-process implementation of the interface that carries its own transport to the node agent, with delegation staying topology-private. That's the pattern the RFC already describes, in the alternatives section and again in non-goals, rather than anything new.
Three things in the contract
1. A transport surface for the delegated link. The alternatives section says a transport-bearing surface "is not precluded: it is versioned contract surface, added when a concrete delegated backend requires it." We'd be that backend, so this is us supplying the trigger rather than reopening the decision. What we'd want to agree is the shape: which operations cross the link, and what the versioning story is when the two halves are deployed separately and can skew.
2. Confirmation evidence as properties rather than mechanisms. SandboxConfirmEvidence::validate currently requires a specific set of seccomp user-notification facts plus landlock_abi >= 3. A gVisor boundary provides neither, because the sentry implements the syscall interception itself, so no backend on it can reach Ready regardless of the isolation it actually provides. The merged RFC doesn't ask for either mechanism, and its kernel-relationship table already contemplates a userspace application kernel, so I read this as a gap between the text and the crate. Written up properly in #3361 rather than here, with the measured probe output and a proposed shape.
3. A memory-preserving lifecycle verb, plus a capability bit. Today's lifecycle is cold: Stop and Start by design. Substrate suspends an actor with its memory intact and resumes it, which is the property the density model depends on, and there's no way to express it through the current verbs. GetCapabilities already negotiates gateway_manages_lifecycle, so negotiating lifecycle semantics per backend is an established pattern here, and a backend that doesn't implement the verb would simply not advertise it.
One thing I'd rather ask than assume: does BoundaryLoopbackConnector cover port forwarding into a delegated topology, or is that still open?
What we'd do
Build it, against a real deployment rather than a mock. We have Substrate running on GKE today with gVisor actors suspending and resuming with memory and filesystem state both intact, so the backend has something to sit on from day one.
We'd also be glad to contribute the delegated-topology scenarios to openshell-conformance as part of the work. A backend that's out of process is the awkward case for a suite, and it seems better for us to write those than to hand you a backend and leave the testing to someone else.
Happy to break any of the three out into its own issue if that's easier to review than one thread.
Activity
mayawang commentedon Sep 16, 2026
@drew this is the proposal half of what you asked for on #2048. #3361 is the narrow piece, on the confirmation evidence itself; this one is the delegated backend surface, the transport, and a lifecycle verb that preserves memory behind a capability bit.
Flagging it because the only pointer to it was a cross-reference. Happy to split it into smaller issues if that reviews better.
github-actions commentedon Oct 2, 2026
This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.
danehans commentedon Oct 7, 2026
Hi all! I'm a +1 to see Substrate as a supported isolation backend. In general, I think it would be helpful to exercise the different OpenShell extension points.
danehans commentedon Oct 8, 2026
This would let OpenShell agents free up compute during idle without losing state, while keeping its policy and credential controls in OpenShell. @drew with #3366 merged, will you be open to accepting a Substrate backend along these lines?
iracic82 commentedon Oct 22, 2026
I've been testing OpenShell with Agent Substrate on kind, mostly to see if we can avoid a supervisor pod per agent (#2385). One result changes the picture here a bit.
The issue describes Substrate isolation as gVisor, and on gVisor I see what #3361 measured: runsc
release-20261012.0gives ENOSYS for Landlock and EINVAL forSECCOMP_FILTER_FLAG_NEW_LISTENER, soopenshell-sandboxfails closed. But Substrate also has a microVM class (ateworker-microvm: Kata 4.1.0 guest kernel 6.18, Cloud Hypervisor), and there the stock runtime qualifies.Inside a Substrate microVM actor (substrate@ab4f3dbe),
openshell-sandbox capability-probe-launch 1000 1000from the 0.1.5 release returns"qualified": truewith Landlock ABI 7. Same result after suspending and resuming the actor, 3 of 3 runs.Two things the actor needs, both on the Substrate side:
net.ipv4.ip_unprivileged_port_start=0. The Kubernetes driver sets it as a pod sysctl, but on Substrate main the guest defaults to 1024. microvm: forward sysctls and lift the guest's privileged-port limit to match gVisor agent-substrate/substrate#1904 fixes that, and with it merged the runtime qualifies with no workaround.runAsUser. The launcher handles that withcapabilities: {drop: [ALL], add: [SETPCAP, SETUID, SETGID]}. With the default capability setPR_CAPBSET_DROPfails with EPERM.So I think the microVM class could use the native backend as it is, and the delegated backend would only be needed for gVisor actors. Maybe that's a smaller first step.
I also ran the shared supervisor from #4702 on Kubernetes (results on that PR). Per sandbox it goes from 2 pods and 16.4 MiB of supervisor memory to 1 pod and about 5.4 MiB. Wiring that to Substrate actors needs a driver path, which is what this issue is about.
@drew is a microVM-first path something you'd consider here? If yes, I'm happy to work on it.
JimBugwadia commentedon Nov 17, 2026
Hi @mayawang @iracic82 - now that agent-substrate is 1.0 what is the next step for this initiative? Thanks!
iracic82 commentedon Nov 17, 2026
Yo Jim, good timing. I re-ran everything on the 1.0 release before answering, so here's where it actually stands.
The microVM path works on 1.0. Substrate v1.0.0, microVM class (Kata 4.3.1, guest kernel 6.18.56), and the stock
openshell-sandboxbinary from the v0.1.8 release:capability-probe-launchcomes back"qualified": truewith Landlock ABI 7 and seccomp notification working. 3 of 3 runs, on a fresh actor and again after suspend and resume. The only thing the template sets iscapabilities: {drop: [ALL], add: [SETPCAP, SETUID, SETGID]}. No sysctl hack anymore, since agent-substrate/substrate#1904 shipped in 1.0, and a counter actor keeps its in-memory state across the suspend.Two 1.0 changes to know about. Containers now run as the image's
USER(agent-substrate/substrate#2392), and the OpenShell launcher has to start as root: an image withUSER 1000fails closed with "capability-free launch must start as root". Easy to catch when the template is created. Andcapabilities.addis now an allowlist (agent-substrate/substrate#3580); the three caps above are all on it.OpenShell moved a lot too since Maya's proposal:
OPENSHELL_COMPUTE_DRIVER_SOCKETisolation_backend.protois now a gRPC service with versionedDescribenegotiation (marked experimental), which is pretty much the delegated transport Maya asked forSo the contract pieces are mostly there. What's missing is the Substrate driver itself.
To be upfront about scope: what I tested is OpenShell's own runtime qualification inside a Substrate actor. I haven't run a full sandbox from the gateway into an actor yet, because that's exactly the driver's job.
What I'd like to do next:
@drew two questions before I start. Are you good with this living out of tree behind the driver socket, or would you rather have it in-tree? And for this placement the driver would own the outer fence: what evidence would you want for default-deny egress and fail-closed controller loss? Substrate has its own egress gateway, but I haven't tested it against your fence guarantees, so I'd rather agree on that before writing code.
JimBugwadia commentedon Nov 17, 2026
Thanks @iracic82 - this is great! Is there an RFC and / or a prototype available to try?
If this is out-of-tree, what would the user install experience be? Separately install substrate and the driver, after installing the OpenShell chart?
On the egress gateway and rules, substrate has an internal policy and there was a proposal on using OpenFGA and allowing external engines. OpenShell also has its own YAML format, and is moving towards Cedar. For an OpenShell user, it would be best to stay with OpenShell defined APIs. How do you see these aligning?