Repository navigation
Description
Summary
NVML/nvidia-smi and Vulkan work inside a --gpu sandbox, but any CUDA workload fails at init with cudaErrorOperatingSystem (304), so RTX/OptiX/ML workloads can't run. Root cause: OpenShell's seccomp profile blocks memfd_create, which the NVIDIA CUDA driver requires during initialization.
Environment
- OpenShell 0.0.53 and 0.0.54 (same behavior)
- Docker driver, GPU via CDI (
nvidia.com/gpu=all) - Ubuntu 24.04, kernel 6.8, NVIDIA driver 570.169, GPU RTX 5080
- Workload: NVIDIA OVRTX (Omniverse RTX) Python SDK
Symptom
[Error] [omni.rtx] CUDA error 304: cudaErrorOperatingSystem - OS call failed or operation not supported on this OS
[Error] [omni.rtx] Failed to query CUDA device count.
[Error] [ovrtx] createDevices failed -> Failed to setup graphics!
Root cause (confirmed)
- GPU is correctly attached — all
/dev/nvidia*nodes are present in the container (docker exec -u 0), and the same workload renders successfully viadocker execinto the same container (bypassing the process-sandbox). So GPU/driver/image are fine. - Seccomp is default-allow with ~20 targeted blocks; CUDA works under Docker's default seccomp, so the culprit is one of those blocks.
- Direct test:
ctypescallingmemfd_create(syscall 319) ->EPERMunder OpenShell, succeeds viadocker exec— matches the block inseccomp.rs(commented re: fileless execution / landlock bypass). - Ruled out: landlock (disabling it didn't help), version (0.0.54 same), LD_PRELOAD (static binary), strace (ptrace blocked).
Why CUDA needs it
The NVIDIA driver/CUDA runtime uses memfd_create during initialization. With it blocked, CUDA can't initialize -> error 304.
Proposed fix (security-preserving), in order of preference
- Allow
memfd_createonly withMFD_NOEXEC_SEAL(kernel >= 6.3; ours is 6.8). Permits non-executable in-memory files (what CUDA needs) while still blocking executable memfds — the fileless-execution vector stays closed. - Or relax the block only when the sandbox is created with
--gpu. - Or expose a policy knob to opt in.
Security assessment
Regression from option (1) is essentially nil — executable memfds remain blocked, and the primary containment (network egress proxy + allowlist, credential isolation, non-root, no ptrace, the other seccomp blocks, landlock FS) is unchanged. memfd_create grants in-memory file creation, not network/secret/host access.
Impact
Unblocks the entire class of CUDA/RTX/ML agent workloads, uniformly across all backends (Docker, Podman, Kubernetes, MicroVM). Happy to validate a patched build, and can attempt a PR.
Activity
elezar commentedon Jun 4, 2026
Thanks for the detailed report. Based on follow-up discussion, this may have been resolved by #1522, which restored the GPU
/procread-write compatibility baseline for CUDA workloads.That change is included in v0.0.55. Could you retest with v0.0.55 or newer and confirm whether CUDA initialization still fails?
One related follow-up is #1628, which tracks narrowing the GPU procfs permission model while keeping runtime-added permissions visible/auditable. That may affect the exact behavior in the future, but the current v0.0.55 behavior should be the right baseline to test against.
elezar commentedon Jul 10, 2026