Repository navigation
Description
User Story
We use OpenShell as the sandbox runtime for NemoClaw. While qualifying the OpenShell 0.1.3 upgrade in NVIDIA/NemoClaw#12603, we encountered two failing Docker GPU compatibility scenarios.
NemoClaw retains compatibility routes for Docker Desktop WSL and Jetson/Tegra. We need a supported GPU provisioning workflow that preserves sandbox security and lifecycle ownership. The real-host evidence below covers ARM64 Docker Desktop WSL only; Jetson/Tegra requirements remain unqualified.
Terminology and Requested Behavior
- OpenShell Runtime owns sandbox creation, supervision, isolation, credentials, and lifecycle.
- NVIDIA Container Runtime is Docker's
nvidiaruntime, selected with--runtime=nvidia. Earlier references to “NVIDIA runtime” meant this component, not OpenShell Runtime. - CDI describes GPU devices and their required resources; OpenShell currently uses NVIDIA CDI discovery for its native GPU path.
In response to elezar's question, the Docker compatibility behaviors under discussion are equivalents of:
docker run --gpus={{ .Request }} ...
docker run --gpus={{ .Request }} --runtime=nvidia ...These describe the host behaviors we want to accommodate, not a requirement to add two new OpenShell modes. On the tested host, --gpus all alone works for GPU visibility with Docker's default runtime runc; explicitly selecting --runtime=nvidia also works but is not necessary for that check. Other host configurations must be tested separately.
We prefer a supported CDI solution if #4204 and #2846 can provide NVIDIA device discovery and the required sandbox access on this host. Whether any additional non-CDI mechanism belongs in OpenShell remains an open maintainer design decision.
Problem Statement
The released OpenShell 0.1.7 Docker GPU path rejects the tested WSL host before sandbox allocation because no NVIDIA CDI devices are discovered, while Docker GPU visibility checks succeed. The evidence below motivates resolving that host gap; it does not establish that an additional compatibility mode is preferable to the ongoing CDI work.
NemoClaw's older compatibility code recreates a workload container externally. That approach does not carry forward to the split supervisor/workload runtime: OpenShell owns bootstrap staging, launch credentials, workload networking, and restart identity.
The original single-container selector was a NemoClaw defect. Beyond selection, NemoClaw's legacy restart/recreation validator also assumes the older container contract and rejects the split workload. Those integration defects belong in NemoClaw; they are separate from the native OpenShell CDI discovery failure reproduced below. Correcting role selection alone does not make external workload recreation safe or resolve missing NVIDIA CDI discovery.
This issue requests a supported runtime workflow. It does not assert that OpenShell has already promised support for every affected host.
Impact / Why This Matters
The upgrade cannot qualify the retained GPU compatibility behavior. The previous full NemoClaw run failed both scenarios:
Those CI results establish the compatibility-path failure on Linux. They are not physical WSL or Jetson validation.
Real-Host Reproduction — 2026-11-16
| Component | Tested configuration |
|---|---|
| Hardware | ARM64 NVIDIA RTX Spark N1X |
| Windows / NVIDIA driver | Windows 28000.3231 / driver 621.14 |
| WSL / Linux | WSL 2.7.14.0; kernel 6.18.33.2-microsoft-standard-WSL2; Ubuntu 24.04.5 |
| Docker | Desktop 4.97.0.244584; Engine 29.9.2; aarch64; default runtime runc |
| OpenShell | Official released 0.1.7 ARM64 package openshell_0.1.7-1_arm64.deb |
| Networking | WSL mirrored networking, unchanged |
OpenShell package SHA256 (matched to release metadata):
354f74db54b21ed34f7bacdc5c97ee35749e4d3690f2c3b83b3a6e32ac5bed05.
Docker discovery fields from docker info --format '{{json .}}':
{
"CDISpecDirs": ["/etc/cdi", "/var/run/cdi"],
"DiscoveredDevices": [
{"Source": "cdi", "ID": "docker.com/gpu=webgpu"}
],
"DefaultRuntime": "runc"
}CDI directories and a WebGPU CDI entry exist, but no NVIDIA CDI device is discovered. This is not a claim that all CDI discovery is empty.
Using the same public ARM64 Ubuntu image for the controls and native creation request:
IMAGE=ubuntu@sha256:73b999393d07f2bf65a40373778a44dc9a6d54ad7718d29247d8ff9413c4d6a8
docker run --rm --network none --gpus all "$IMAGE" \
nvidia-smi --query-gpu=name,driver_version --format=csv,noheader
docker run --rm --network none --gpus all --runtime=nvidia "$IMAGE" \
nvidia-smi --query-gpu=name,driver_version --format=csv,noheaderBoth exit 0 and print:
NVIDIA RTX Spark N1X (6144-core Blackwell RTX GPU), 621.14
Against an isolated, authenticated OpenShell 0.1.7 gateway:
openshell --gateway TASK_GATEWAY sandbox create \
--name TASK_SANDBOX --from "$IMAGE" --gpu --detach -- sleep 600This exits 1 in approximately 0.098 seconds:
no NVIDIA CDI GPU devices were discovered
The gateway log records RPC status 9 rejecting creation before allocation. The gateway used mTLS user authentication and gateway-issued sandbox JWTs; no insecure flags, external container cloning, bootstrap replay, generated CDI specs, or daemon/network configuration changes were used. Task-owned runtime resources were cleaned up.
Evidence limits: the Docker controls prove GPU visibility, not CUDA kernel execution or inference. Native OpenShell creation did not reach workload execution, so this does not qualify sandbox CUDA, lifecycle, endpoint connectivity, or NemoClaw E2E behavior. Jetson and x86 WSL remain untested. Although this host uses mirrored networking, this failure happens before the seccomp listener path in #4902; it neither reproduces nor resolves that separate issue.
Proposed Design
A caller should request a GPU sandbox through OpenShell's managed creation workflow. First determine whether the existing CDI work can support the observed host configuration. If an additional GPU injection mechanism is needed, its necessity and supported host contract require maintainer agreement before implementation. OpenShell must retain lifecycle ownership.
GPU attachment alone is insufficient. Landlock policies must permit the required device nodes and libraries, with the CDI-derived policy work in #4204 considered as part of the solution. The workflow must preserve required devices, libraries, and supplementary groups together with supervisor/workload identity, bootstrap credentials, networking, restart, rollback, and cleanup. Non-root execution, seccomp, dropped capabilities, and network isolation must remain enforced.
Please establish the supported host requirements and configuration contract before implementation. This proposal does not require a particular new CLI flag or internal replacement mechanism.
Scope and Ownership
- WSL discovery/provisioning: the real-host gap above is for OpenShell maintainers to assess alongside #4849 / #4854 and the CDI work. It is not evidence of an accepted non-CDI design.
- Jetson/Tegra: no physical reproduction or qualification is supplied here. A separately scoped follow-up needs host discovery/error evidence and maintainer agreement; no new Jetson issue is claimed to exist.
- NemoClaw: repair its legacy selector/recreation assumptions and qualify the retained scenarios against a published, supported OpenShell artifact in NVIDIA/NemoClaw#12603. OpenShell feature acceptance does not itself complete that qualification.
- Mirrored networking: track separately in Kubernetes driver on WSL2 with mirrored networking: sandbox start fails with
seccomp notification probe: Resource busy (os error 16)(follow-up to #3842) #4902; successful GPU provisioning would still leave that execution path to test.
The broader acceptance criteria below remain outstanding. This description separates the workstreams in response to triage; it does not claim that separate deliverables have been accepted or completed.
Acceptance Criteria
- Establish a supported OpenShell workflow for the existing Docker GPU compatibility requirements, including hosts without usable NVIDIA CDI devices, or demonstrate an equivalent CDI path on those exact host configurations.
- Demonstrate real CUDA initialization through that workflow on Docker Desktop WSL and Jetson/Tegra. Record runtime versions, host profiles, tested commits, commands, and results.
- Preserve the selected GPU devices, required libraries, supplementary groups, and inference behavior through sandbox stop/start and gateway restart.
- Keep workload networking isolated and keep launch credentials under runtime ownership. Callers must not copy bootstrap private keys or externally clone the workload to obtain GPU access.
- On creation or transition failure, preserve existing workload data and rollback authority. Reject foreign or ambiguous workload identities.
- Qualify both NemoClaw compatibility-only and native-rejection/fallback scenarios using the published runtime artifact. Preserve the existing assertions.
Alternatives Considered
- CDI-first solution: feat(gpu): derive Docker sandbox requirements from CDI specs #4204 and fix(supervisor): preserve standard sbin paths in sandbox shells #2846 are the maintainer-identified work to assess before adding compatibility behavior. The historical assessment of feat(gpu): derive Docker sandbox requirements from CDI specs #4204 at
33350a881f55011286e7c3a68d8007e43e50e0bfcovered CDI-derived devices, mounts, sockets, and groups; it was not a qualification of the current revisions on the host above. We have not established that those proposals cannot solve this case. - Existing design: #1606 covers CDI-derived sandbox access. This issue records the remaining compatibility requirement rather than duplicating that resolver work.
- External container recreation: fixing role selection leaves bootstrap staging, network isolation, and restart admission unresolved.
- Middleware or inference providers: these extension points do not own Docker GPU device injection or workload creation.
- Disable GPU support: that does not preserve NemoClaw's retained support requirements.
Agent Investigation
Historical source inspection used OpenShell 0.1.3 revision e1f3c82caa3ed3b65de22889ae7ef32a774878ef and release tag v0.1.5. These are historical references, not the release used for the new reproduction above.
In the v0.1.5 Docker driver, DockerSandboxDriverConfig exposes cdi_devices and mounts, and GPU DeviceRequest uses driver cdi. Source inspection does not establish a live failure on v0.1.5.
The 0.1.3 driver stages runtime/bootstrap files after workload creation and requires workload network mode none. Restart selection cannot target an externally created replacement by immutable Docker ID while an original with the same sandbox labels remains retained.
The new physical ARM64 WSL reproduction is recorded above. Jetson remains untested. Separately, rsliter's earlier prototype report describes CUDA execution, isolation, and lifecycle testing on ARM64 N1X WSL using an unpublished prototype based on #4204. That attributed prototype evidence is not released OpenShell 0.1.7 behavior, Jetson/x86 WSL qualification, or completion of NemoClaw's compatibility E2E scenarios.
Checklist
- Reviewed relevant existing issues, PRs, runtime source, and repository issue templates.
- This is a workflow proposal with observable acceptance criteria.
Activity
rsliter commentedon Oct 22, 2026
We have a local prototype extending #4204 with OpenShell-owned NVIDIA GPU injection for Docker Desktop WSL, tested on real ARM64 N1X hardware. This provides a path to unblock the most critical flow in this issue for the upgrade to OpenShell 0.1.x.
What it does
--gpu --driver-config-json '{"docker":{"nvidia_wsl_gpus":"all"}}'. The gateway operator must enable driver configuration.What we tested
On an NVIDIA RTX Spark N1X running Ubuntu 24.04 under WSL2, Docker Desktop 4.96.0 / Engine 29.9.1, and NVIDIA driver 620.66:
The Docker driver suite passed 165 tests on both macOS and ARM64 Linux, with two existing tests ignored. Strict Clippy and formatting checks also passed.
Scope and remaining work
This is an unpublished prototype based on #4204 at
33350a881f55011286e7c3a68d8007e43e50e0bf. It demonstrates the explicit all-GPU path on this ARM64 WSL host. It does not yet qualify Jetson, x86 WSL, individual GPU selection on multi-GPU hosts, model inference, or NemoClaw’s two compatibility scenarios against a published runtime artifact.The hardware test used an isolated development gateway configuration, so production gateway transport/authentication is also outside this evidence.
This supports a focused Docker Desktop WSL implementation slice; it does not establish completion of all #4837 acceptance criteria.
Feedback or Approval Requested
We'd love feedback on this approach, or approval to proceed.
purp commentedon Nov 5, 2026
Thanks for the detailed write-up. The problem is clear: OpenShell's Docker driver does not support the Docker Desktop WSL and Jetson/Tegra configurations that NemoClaw needs. A few specifics are needed before a maintainer can decide.
What the current code does. On
main, the Docker driver treats CDI as supported whendocker inforeports non-emptyCDISpecDirs. It builds its GPU inventory only fromDiscoveredDevicesentries with sourcecdi. The only injection path is aDeviceRequestwith drivercdi. If no NVIDIA CDI devices are discovered,--gpufails with "no NVIDIA CDI GPU devices were discovered". If the daemon reports no CDI spec dirs, it fails with "docker GPU sandboxes require Docker CDI support".The existing WSL2
nvidia.com/gpu=allfallback applies only when the daemon already listsnvidia.com/gpu=allas a discovered device. It does not help when discovery returns nothing. This is consistent with the issue's premise, but the code alone cannot show what Docker Desktop WSL and Jetson hosts actually report.Information needed
docker infoCDI fields (CDISpecDirs,DiscoveredDevices), plus the exact error fromopenshell sandbox create --gpuon currentmainor the latest release.Scope. This issue combines two host families, hardware qualification, rollback guarantees, and cross-repo NemoClaw qualification. Please split it into separately deliverable issues, each with its own acceptance criteria:
Then narrow or retitle this issue to what remains. Related: #4204 (CDI base, currently open and stale), #1606 (closed), #4327.
Next human decision: a maintainer decides whether to accept the split work once the above is provided.
elezar commentedon Nov 5, 2026
@prekshivyas could you please update the issue description to make the distinction betwen the OpenShell Runtime and the NVIDIA Container Runtime (which I assume is referred to as the "NVIDIA runtime") more clear.
If I understand that ask, what you are asking for is a mechanism to switch between our current CDI-only-based injection to at least the following alternatives:
docker run --gpus={{ .Request }}docker run --gpus={{ .Request }} --runtime=nvidiaIs this correct?
In general, I don't think pushing the compatibility behaviour into OpenShell is the right approach here (or at least not without further motivation). The existing work in #4204 and #2846 attempts to address platform-specific inconsistencies and should remove the need for most of the workarounds that are being put in place on these systems.
One thing to note: Injecting a GPU into the sandbox using an alternative mechansim does not guarantee that the injected paths (device nodes, libraries, etc.) are accessible in a sandbox. These may still be restricted by the landlocked policies that we're applying. The intent of #4204 is to ensure that we also use the CDI specification to populate the required sandbox policies.
fix(onboard): prepare NVIDIA CDI for Docker Desktop WSL (#14458)
prekshivyas commentedon Nov 16, 2026
@elezar Updated the issue description in place to address your feedback:
--gpus alland--gpus all --runtime=nvidiapass GPU visibility checks, while released OpenShell 0.1.7 rejects creation withno NVIDIA CDI GPU devices were discovered. Docker reports a WebGPU CDI entry, but no NVIDIA CDI devices; its default runtime isrunc.Does this provide the motivation and host evidence you needed to assess whether the existing CDI work covers this case?