diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index 59a7d15b1d..34e3f36094 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -447,8 +447,19 @@ Use `mise run e2e:kubernetes` for the standard Helm-backed Kubernetes suite. Set `OPENSHELL_E2E_KUBE_TEST=kubernetes_image_update` to scope the Rust suite to combined image, command, environment, and metadata updates with retained sandbox, SSH, and PVC identities. The next start verifies the new argv and environment. This target is -included in both existing Agent Sandbox API-version lanes; unlike the Python -Docker suite, it exercises the driver that supports image updates. +included in both existing Agent Sandbox API-version lanes. Docker separately +supports image-only updates on stopped, opt-in persistent `/sandbox` workspaces; +this Kubernetes lane does not qualify that path. Docker requires locally cached +images, a matching numeric target `USER UID:GID`, and no image-declared volumes. +Do not treat a skipped Docker lane as live image-replacement coverage. + +The separate `docker-updates` testkit suite qualifies the full Docker path +through an owned mTLS gateway, including SSH exec, image replacement, restart, +credential-publication repair and deletion. Follow +`tests/suites/features/docker-updates/README.md`; the wrapper's +`OPENSHELL_E2E_DOCKER_IMAGE_UPDATES=1` opts in only its fresh gateway and refuses +an existing endpoint. On Docker Desktop without host networking, select the +documented bridge override rather than changing engine settings. Image updates default to disabled outside test fixtures. The e2e wrapper opts in only for a fresh gateway fixture that it owns; set diff --git a/crates/openshell-driver-docker/README.md b/crates/openshell-driver-docker/README.md index f84d792910..268dca4f4b 100644 --- a/crates/openshell-driver-docker/README.md +++ b/crates/openshell-driver-docker/README.md @@ -128,8 +128,89 @@ New generations require matching supervisor and sandbox runtime binaries. ## Stop, Start, and Delete +Set `persist_workspace = true` in `[openshell.drivers.docker]` to give newly +created default `/sandbox` workspaces a driver-owned local volume. The setting +defaults to false and does not migrate existing writable-layer workspaces. +Custom working directories and image-declared volumes overlapping `/sandbox` +are rejected when enabled. Initialization prepares the workspace ownership; +it does not copy the image's workspace contents into the volume. + +The driver records and rechecks the volume identity and workload UID/GID before +restart, including when external resource admission is disabled. Stop retains +the data. Explicit sandbox deletion removes the owned workspace volume and +reports removal failures. Startup orphan cleanup retains workspace volumes +without a workload; an interrupted create requires explicit recovery/removal. +This option advertises the optional workload update capability. The gateway's +separate `enable_sandbox_updates` setting must also be enabled to accept updates. + +Persistence ownership comes from `workspace-storage.json` in the protected +per-sandbox driver directory, bound into the private bootstrap and runtime +descriptor. It records the namespace, sandbox, workload UID/GID and exact volume +identity. Container labels do not select persistent storage or authorize volume +admission, updates or removal. All caller template labels retain their existing +meaning, including generic and Docker-specific workspace label names. + +### Stopped Workload Image Updates + +The Docker updater implements the existing configuration-update RPC for +image-only changes on stopped, persistent `/sandbox` workspaces. Source and +target images must already be cached locally and declare no image volumes. The +target must declare an explicit numeric `USER UID:GID` matching the recorded +non-root workload identity. Symbolic users/groups, UID-only users, and an absent +`USER` are rejected before replacement; update preflight does not resolve image +accounts. Command and environment changes are unsupported. + +The update replaces the stopped workload container while preserving the native +sandbox ID and owned workspace volume. It refreshes the private runtime evidence +for the replacement; it does not change the trusted sandbox or supervisor images. +A private operation record lets the same accepted operation resume after an +interruption. Start refuses an unfinished update. Stop and startup reconciliation +retain its data and runtime resources; explicit Delete removes its owned storage. +Retry through the gateway to finish the accepted update before starting it. +The operation never starts the workload or rolls back workspace data implicitly. + +#### Live Driver Integration Test + +The opt-in test below uses a real Docker daemon and isolated, uniquely labeled +stopped containers and a workspace volume. It checks numeric/symbolic identity +refusal, A-to-B replacement, retained file bytes, replay, and recovery after an +accepted update loses its source container. Cleanup verifies ownership and never +forces removal. This is driver integration coverage; gateway authorization, +supervisor startup, and authenticated workload operations still require E2E. + +From the repository root, prepare the four local test images: + +```shell +for version in A B BAD SYMBOLIC; do + case "$version" in + BAD) fixture_user=2000:2000 ;; + SYMBOLIC) fixture_user=other:other ;; + *) fixture_user=1234:1235 ;; + esac + docker build --network=none --build-arg VERSION="$version" \ + --build-arg WORKLOAD_USER="$fixture_user" -t "openshell-image-update-test:$version" \ + crates/openshell-driver-docker/tests/fixtures/image-update +done + +OPENSHELL_DOCKER_UPDATE_IMAGE_A=openshell-image-update-test:A \ +OPENSHELL_DOCKER_UPDATE_IMAGE_B=openshell-image-update-test:B \ +OPENSHELL_DOCKER_UPDATE_IMAGE_BAD=openshell-image-update-test:BAD \ +OPENSHELL_DOCKER_UPDATE_IMAGE_SYMBOLIC=openshell-image-update-test:SYMBOLIC \ +cargo test --locked -p openshell-driver-docker --lib \ + live_stopped_image_replacement -- --ignored --nocapture +``` + +The Docker CLI must be on `PATH` and use the same daemon as the SDK. On Docker +Desktop, set `DOCKER_HOST` to the socket reported by `docker context inspect`. + +For authenticated gateway/supervisor qualification, including SSH exec, +replacement, gateway restart, interrupted credential-publication repair and +complete API deletion, run the separate +[Docker update testkit suite](../../tests/suites/features/docker-updates/README.md). + Stop terminates the supervisor companion, allowing it 10 seconds to stop the -workload, and stops the workload container without removing it. Docker retains the workload writable layer and attached +workload, and stops the workload container without removing it. Docker retains +the workload writable layer and attached volumes. Start stages a fresh sandbox bootstrap bundle, restarts that workload, and creates a new supervisor companion. A durably stopped sandbox stays stopped across gateway restarts. diff --git a/crates/openshell-driver-docker/src/image_update.rs b/crates/openshell-driver-docker/src/image_update.rs new file mode 100644 index 0000000000..4c5070f2c9 --- /dev/null +++ b/crates/openshell-driver-docker/src/image_update.rs @@ -0,0 +1,1022 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Stopped workload replacement. The gateway remains the accepted-operation owner. + +use super::*; +use bollard::models::ContainerInspectResponse; +use openshell_core::proto::compute::v1::{ + SandboxUpdatePreconditions, UpdateSandboxConfigurationRequest, +}; +use serde::{Deserialize, Serialize}; +use std::io::Write as _; + +const RECORD_FILE: &str = "workload-image-update.json"; +const OPERATION_LABEL: &str = "openshell.ai/workload-image-operation"; +const REVISION_LABEL: &str = "openshell.ai/workload-image-revision"; +const MAX_RECORD_BYTES: usize = 2 * 1024 * 1024; + +#[derive(Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +enum Phase { + Prepared, + Replaced, + Applied, +} + +// No Debug implementation: the immutable create input can contain private +// environment/configuration. It never enters logs or public status. +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Intent { + version: u32, + namespace: String, + sandbox_id: String, + sandbox_name: String, + operation_id: String, + revision: u64, + previous_image: String, + image: String, + image_id: String, + original_container: String, + container_name: String, + target_container: Option, + body: ContainerCreateBody, + original_bootstrap: Vec, + original_descriptor: Vec, + phase: Phase, +} + +fn invalid(message: &'static str) -> Status { + Status::failed_precondition(message) +} + +fn validate_request( + request: &UpdateSandboxConfigurationRequest, +) -> Result<&SandboxUpdatePreconditions, Status> { + let p = request + .preconditions + .as_ref() + .ok_or_else(|| Status::invalid_argument("update preconditions are required"))?; + if p.sandbox_id.is_empty() + || p.sandbox_id.len() > 128 + || !p + .sandbox_id + .bytes() + .all(|v| v.is_ascii_alphanumeric() || matches!(v, b'-' | b'_')) + || p.name.is_empty() + || p.name.len() > 256 + || p.image.is_empty() + || p.image.len() > 4096 + || p.expected_image.is_empty() + || p.expected_image.len() > 4096 + || p.revision == 0 + { + return Err(Status::invalid_argument( + "invalid Docker image-update preconditions", + )); + } + let operation = p.operation_id.as_bytes(); + if operation.len() != 36 + || operation.iter().enumerate().any(|(i, v)| { + if matches!(i, 8 | 13 | 18 | 23) { + *v != b'-' + } else { + !v.is_ascii_hexdigit() + } + }) + { + return Err(Status::invalid_argument( + "image-update operation ID must be a UUID", + )); + } + if request.main_process.is_some() || request.environment.is_some() { + return Err(Status::unimplemented( + "the initial Docker updater supports image-only workload changes", + )); + } + // Docker's pinned contract does not authenticate compute credentials or + // publish that optional identity. The private native descriptor, exact + // container ID and recorded persistent-volume identity anchor this update. + if !p.expected_runtime_identity.is_empty() { + return Err(invalid( + "Docker image update does not recognize the supplied compute runtime identity", + )); + } + Ok(p) +} + +fn path(config: &DockerDriverRuntimeConfig, id: &str) -> Result { + Ok(docker_boundary_state_dir_by_id(id, config)?.join(RECORD_FILE)) +} + +fn read(config: &DockerDriverRuntimeConfig, id: &str) -> Result, Status> { + let path = path(config, id)?; + let metadata = match std::fs::symlink_metadata(&path) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(_) => return Err(invalid("Docker image-update record cannot be read")), + }; + if !metadata.is_file() || metadata.len() > MAX_RECORD_BYTES as u64 { + return Err(invalid( + "Docker image-update record is not a bounded regular file", + )); + } + let intent: Intent = serde_json::from_slice( + &std::fs::read(path).map_err(|_| invalid("Docker image-update record cannot be read"))?, + ) + .map_err(|_| invalid("Docker image-update record is invalid"))?; + if intent.version != 1 + || intent.sandbox_id != id + || intent.namespace != config.sandbox_namespace + { + return Err(invalid( + "Docker image-update record belongs to another native resource", + )); + } + Ok(Some(intent)) +} + +pub fn durable_write(path: &Path, contents: &[u8]) -> Result<(), Status> { + let parent = path + .parent() + .ok_or_else(|| invalid("Docker image-update state directory is missing"))?; + let mut temporary = tempfile::NamedTempFile::new_in(parent) + .map_err(|_| Status::internal("create private Docker image-update checkpoint"))?; + temporary + .write_all(contents) + .and_then(|()| temporary.as_file().sync_all()) + .map_err(|_| Status::internal("flush Docker image-update checkpoint"))?; + temporary + .persist(path) + .map_err(|_| Status::internal("publish Docker image-update checkpoint"))?; + std::fs::File::open(parent) + .and_then(|file| file.sync_all()) + .map_err(|_| Status::internal("flush Docker image-update directory")) +} + +fn save(config: &DockerDriverRuntimeConfig, intent: &Intent) -> Result<(), Status> { + let bytes = serde_json::to_vec(intent) + .map_err(|_| Status::internal("encode Docker image-update checkpoint"))?; + if bytes.len() > MAX_RECORD_BYTES { + return Err(invalid("Docker image-update checkpoint exceeds its bound")); + } + durable_write(&path(config, &intent.sandbox_id)?, &bytes) +} + +fn same_request(intent: &Intent, p: &SandboxUpdatePreconditions) -> bool { + (intent.sandbox_id.as_str(), intent.sandbox_name.as_str()) + == (p.sandbox_id.as_str(), p.name.as_str()) + && intent.operation_id == p.operation_id + && intent.revision == p.revision + && intent.previous_image == p.expected_image + && intent.image == p.image +} + +fn require_stopped(container: &ContainerInspectResponse) -> Result<(), Status> { + if container.state.as_ref().is_none_or(|state| { + state.running != Some(false) + || state.paused != Some(false) + || state.restarting != Some(false) + || !matches!( + state.status, + Some(ContainerStateStatusEnum::CREATED | ContainerStateStatusEnum::EXITED) + ) + }) { + return Err(invalid( + "Docker workload must be verifiably stopped before image replacement", + )); + } + validate_docker_outer_fence(container)?; + let host = container + .host_config + .as_ref() + .ok_or_else(|| invalid("Docker workload launch fence is missing"))?; + if host.privileged == Some(true) + || host + .cap_add + .as_ref() + .is_some_and(|values| !values.is_empty()) + || host + .cap_drop + .as_ref() + .is_none_or(|values| !values.iter().any(|value| value.eq_ignore_ascii_case("all"))) + || host.security_opt.as_ref().is_none_or(|values| { + !values.iter().any(|value| { + matches!( + value.as_str(), + "no-new-privileges" | "no-new-privileges:true" + ) + }) + }) + || host.auto_remove == Some(true) + { + return Err(invalid("Docker workload launch fence changed")); + } + Ok(()) +} + +fn verify_replacement(container: &ContainerInspectResponse, intent: &Intent) -> Result<(), Status> { + let actual = container + .config + .as_ref() + .ok_or_else(|| invalid("Docker replacement configuration is missing"))?; + let expected = &intent.body; + if container.image.as_ref() != Some(&intent.image_id) + || actual.user != expected.user + || actual.entrypoint != expected.entrypoint + || actual.cmd != expected.cmd + || actual.working_dir != expected.working_dir + || intent + .target_container + .as_ref() + .is_some_and(|id| container.id.as_ref() != Some(id)) + || expected.labels.as_ref().is_none_or(|labels| { + labels.iter().any(|(key, value)| { + actual.labels.as_ref().and_then(|labels| labels.get(key)) != Some(value) + }) + }) + { + return Err(invalid( + "Docker replacement does not match the accepted immutable input", + )); + } + for entry in expected.env.as_deref().unwrap_or_default() { + let (key, _) = entry + .split_once('=') + .ok_or_else(|| invalid("Docker bootstrap environment is invalid"))?; + let matches = actual + .env + .as_deref() + .unwrap_or_default() + .iter() + .filter(|value| value.split_once('=').is_some_and(|(name, _)| name == key)) + .collect::>(); + if matches.len() != 1 || matches[0] != entry { + return Err(invalid("Docker replacement bootstrap environment changed")); + } + } + let original_host = serde_json::to_value(&expected.host_config) + .map_err(|_| invalid("accepted Docker launch fence is invalid"))?; + let actual_host = serde_json::to_value(&container.host_config) + .map_err(|_| invalid("Docker replacement launch fence is invalid"))?; + for key in [ + "NetworkMode", + "CapDrop", + "CapAdd", + "Privileged", + "SecurityOpt", + "ReadonlyRootfs", + "Binds", + "Mounts", + "GroupAdd", + "Devices", + "DeviceRequests", + "PidMode", + "UsernsMode", + "IpcMode", + "VolumesFrom", + "AutoRemove", + "Sysctls", + "Tmpfs", + "Dns", + "DnsSearch", + "ExtraHosts", + "Memory", + "NanoCpus", + "PidsLimit", + ] { + if original_host[key] != actual_host[key] { + return Err(invalid("Docker replacement launch settings changed")); + } + } + Ok(()) +} + +fn read_published_boundary( + intent: &Intent, + config: &DockerDriverRuntimeConfig, +) -> Result<(BoundaryConfig, SandboxRuntimeDescriptor), Status> { + let directory = docker_boundary_state_dir_by_id(&intent.sandbox_id, config)?; + let bootstrap: BoundaryConfig = serde_json::from_slice( + &std::fs::read(directory.join(BOUNDARY_CONFIG_FILE)) + .map_err(|_| invalid("replacement Docker bootstrap is unavailable"))?, + ) + .map_err(|_| invalid("replacement Docker bootstrap is invalid"))?; + let descriptor: SandboxRuntimeDescriptor = serde_json::from_slice( + &std::fs::read(directory.join(RUNTIME_DESCRIPTOR_FILE)) + .map_err(|_| invalid("replacement Docker descriptor is unavailable"))?, + ) + .map_err(|_| invalid("replacement Docker descriptor is invalid"))?; + Ok((bootstrap, descriptor)) +} + +fn verify_boundary_resource_bindings( + intent: &Intent, + bootstrap: &BoundaryConfig, + descriptor: &SandboxRuntimeDescriptor, +) -> Result<(), Status> { + let target = intent + .target_container + .as_ref() + .ok_or_else(|| invalid("Docker replacement target is missing"))?; + let (_, expected) = decode_boundary(intent, target)?; + let expected: SandboxRuntimeDescriptor = serde_json::from_slice(&expected) + .map_err(|_| invalid("accepted Docker descriptor is invalid"))?; + // A launch can be interrupted between these files' atomic writes. Each + // must still identify the accepted resource, even when their launch-scoped + // generation/session differ. Only stopped Start with fresh auth repairs it. + for (id, generation, claims, identity, evidence) in [ + ( + &bootstrap.boundary_id, + &bootstrap.generation, + &bootstrap.resource_claims, + &bootstrap.workload_identity, + &bootstrap.provisioning_evidence, + ), + ( + &descriptor.boundary_id, + &descriptor.generation, + &descriptor.resource_claims, + &descriptor.workload_identity, + &descriptor.provisioning_evidence, + ), + ] { + if id != &intent.sandbox_id + || claims != &expected.resource_claims + || identity != &expected.workload_identity + || evidence + != &isolation::provisioning_evidence(target, generation) + .map_err(|_| invalid("replacement Docker evidence is invalid"))? + { + return Err(invalid("replacement Docker resource bindings changed")); + } + } + Ok(()) +} + +fn verify_published_boundary( + intent: &Intent, + config: &DockerDriverRuntimeConfig, +) -> Result<(), Status> { + let (bootstrap, descriptor) = read_published_boundary(intent, config)?; + verify_boundary_resource_bindings(intent, &bootstrap, &descriptor)?; + if bootstrap.generation != descriptor.generation + || bootstrap.session_id != descriptor.session_id + || bootstrap.provisioning_evidence != descriptor.provisioning_evidence + { + return Err(invalid( + "replacement Docker runtime evidence is inconsistent", + )); + } + Ok(()) +} + +fn require_owner( + container: &ContainerInspectResponse, + p: &SandboxUpdatePreconditions, + config: &DockerDriverRuntimeConfig, +) -> Result<(), Status> { + let labels = container + .config + .as_ref() + .and_then(|value| value.labels.as_ref()) + .ok_or_else(|| invalid("Docker image update lacks native ownership metadata"))?; + for (key, value) in [ + (LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE), + (LABEL_SANDBOX_NAMESPACE, config.sandbox_namespace.as_str()), + (LABEL_SANDBOX_ID, p.sandbox_id.as_str()), + (LABEL_SANDBOX_NAME, p.name.as_str()), + (LABEL_ISOLATION_ROLE, LABEL_ISOLATION_ROLE_SANDBOX), + ] { + if labels.get(key).map(String::as_str) != Some(value) { + return Err(invalid( + "Docker image update refused a changed resource owner or storage layout", + )); + } + } + require_stopped(container) +} + +fn decode_boundary(intent: &Intent, target_id: &str) -> Result<(Vec, Vec), Status> { + let mut bootstrap: BoundaryConfig = serde_json::from_slice(&intent.original_bootstrap) + .map_err(|_| invalid("saved Docker bootstrap is invalid"))?; + let mut descriptor: SandboxRuntimeDescriptor = + serde_json::from_slice(&intent.original_descriptor) + .map_err(|_| invalid("saved Docker runtime descriptor is invalid"))?; + if bootstrap.boundary_id != intent.sandbox_id + || descriptor.boundary_id != intent.sandbox_id + || bootstrap.generation != descriptor.generation + || bootstrap.resource_claims.get("docker.container_id") != Some(&intent.original_container) + || descriptor.resource_claims != bootstrap.resource_claims + || descriptor.workload_identity != bootstrap.workload_identity + || !descriptor + .resource_claims + .contains_key(workspace::RECORD_CLAIM) + { + return Err(invalid( + "Docker runtime evidence does not identify the original workload", + )); + } + let identity = &descriptor.workload_identity; + let identity = ResolvedWorkloadIdentity::new( + identity.uid, + identity.gid, + identity.supplementary_gids.clone(), + identity.source.clone(), + intent.image_id.clone(), + ) + .map_err(|_| invalid("Docker workload identity cannot be retained"))?; + let evidence = isolation::provisioning_evidence(target_id, &bootstrap.generation) + .map_err(|_| invalid("Docker replacement provisioning evidence is invalid"))?; + for claims in [ + &mut bootstrap.resource_claims, + &mut descriptor.resource_claims, + ] { + claims.insert("docker.container_id".into(), target_id.into()); + claims.insert("docker.image_identity".into(), intent.image_id.clone()); + } + bootstrap.workload_identity = identity.clone(); + descriptor.workload_identity = identity; + bootstrap.provisioning_evidence = evidence.clone(); + descriptor.provisioning_evidence = evidence; + Ok(( + bootstrap + .encode() + .map_err(|_| Status::internal("encode replacement bootstrap"))?, + serde_json::to_vec(&descriptor) + .map_err(|_| Status::internal("encode replacement descriptor"))?, + )) +} + +impl DockerComputeDriver { + /// Stopped image-only mutation for explicitly enabled persistent workspaces. + /// The gateway owns acceptance, retries and public lifecycle state. + pub async fn update_workload_image( + &self, + request: &UpdateSandboxConfigurationRequest, + ) -> Result<(), Status> { + let p = validate_request(request)?; + if self.config.workspace_storage != workspace::Storage::Persistent { + return Err(Status::unimplemented( + "Docker persistent workspace image updates are not enabled", + )); + } + self.verify_network_trust()?; + let _guard = self + .image_update_gates + .gate_for(&p.sandbox_id) + .lock_owned() + .await; + let previous = read(&self.config, &p.sandbox_id)?; + let mut intent = match previous { + Some(intent) if same_request(&intent, p) => intent, + Some(intent) if intent.phase != Phase::Applied || intent.revision >= p.revision => { + return Err(invalid( + "a conflicting or newer Docker image update is already recorded", + )); + } + _ => Box::pin(self.prepare_image_update(p)).await?, + }; + // A saved operation must retain independent storage authority even if + // interruption removed its original container before replay. + let claim = workspace::record_claim(&p.sandbox_id, &self.config)? + .ok_or_else(|| invalid("private Docker workspace record is missing"))?; + let descriptor: SandboxRuntimeDescriptor = + serde_json::from_slice(&intent.original_descriptor) + .map_err(|_| invalid("saved Docker runtime descriptor is invalid"))?; + if descriptor.resource_claims.get(workspace::RECORD_CLAIM) != Some(&claim) { + return Err(invalid( + "private Docker workspace record and accepted operation disagree", + )); + } + let _ = decode_boundary(&intent, "validation-only-target")?; + if p.validate_only { + return self.validate_update_resource(&intent, p).await; + } + if intent.phase == Phase::Applied { + return self.validate_update_resource(&intent, p).await; + } + save(&self.config, &intent)?; + Box::pin(self.replace_image_container(&mut intent, p)).await + } + + async fn prepare_image_update(&self, p: &SandboxUpdatePreconditions) -> Result { + let directory = docker_boundary_state_dir_by_id(&p.sandbox_id, &self.config)?; + let descriptor_bytes = std::fs::read(directory.join(RUNTIME_DESCRIPTOR_FILE)) + .map_err(|_| invalid("Docker image update requires the original native descriptor"))?; + let descriptor: SandboxRuntimeDescriptor = serde_json::from_slice(&descriptor_bytes) + .map_err(|_| invalid("Docker native descriptor is invalid"))?; + if descriptor.boundary_id != p.sandbox_id { + return Err(invalid( + "Docker native descriptor belongs to another sandbox", + )); + } + let source_id = descriptor + .resource_claims + .get("docker.container_id") + .ok_or_else(|| invalid("Docker native descriptor lacks its container identity"))?; + let source = self + .docker + .inspect_container(source_id, None) + .await + .map_err(|_| invalid("the original Docker workload is unavailable"))?; + if source.id.as_ref() != Some(source_id) { + return Err(invalid( + "Docker inspection did not return the original immutable container ID", + )); + } + require_owner(&source, p, &self.config)?; + self.admit_container_resources(&source).await?; + let original_image = self + .docker + .inspect_image(&p.expected_image) + .await + .map_err(|_| invalid("the expected original Docker image must be present locally"))?; + if original_image.id != source.image + || descriptor.resource_claims.get("docker.image_identity") != source.image.as_ref() + { + return Err(invalid( + "Docker workload image no longer matches the expected original", + )); + } + let target = self.docker.inspect_image(&p.image).await.map_err(|_| { + invalid("the target Docker image must be prepared locally before update") + })?; + let image_id = target + .id + .ok_or_else(|| invalid("target Docker image lacks immutable identity"))?; + let image_config = target + .config + .ok_or_else(|| invalid("target Docker image lacks configuration"))?; + if driver_mounts::resolve_oci_workspace_root( + image_config.working_dir.as_deref().unwrap_or_default(), + ) + .map_err(Status::failed_precondition)? + != "/sandbox" + || image_config + .volumes + .as_ref() + .is_some_and(|volumes| !volumes.is_empty()) + { + return Err(invalid( + "target Docker image requires an unsupported workspace or image volume layout", + )); + } + let source_config = source + .config + .as_ref() + .ok_or_else(|| invalid("Docker workload configuration is missing"))?; + if source_config + .volumes + .as_ref() + .is_some_and(|volumes| !volumes.is_empty()) + { + return Err(invalid( + "original Docker image has anonymous-volume state outside the initial update layout", + )); + } + let expected_user = format!( + "{}:{}", + descriptor.workload_identity.uid, descriptor.workload_identity.gid + ); + if source_config.user.as_deref() != Some(expected_user.as_str()) { + return Err(invalid( + "Docker native descriptor and workload user disagree", + )); + } + // The update RPC carries no identity-policy override. Admit only an + // explicit numeric USER matching the recorded non-root identity; + // symbolic or implicit users need image account resolution, which the + // initial image-only updater does not perform. + let target_identity = image_config.user.as_deref().and_then(|user| { + let (uid, gid) = user.split_once(':')?; + if uid.is_empty() + || gid.is_empty() + || !uid + .bytes() + .chain(gid.bytes()) + .all(|byte| byte.is_ascii_digit()) + { + return None; + } + Some((uid.parse::().ok()?, gid.parse::().ok()?)) + }); + if target_identity + != Some(( + descriptor.workload_identity.uid, + descriptor.workload_identity.gid, + )) + { + return Err(invalid( + "target Docker image must declare matching numeric non-root UID:GID", + )); + } + let mut body: ContainerCreateBody = serde_json::from_value( + serde_json::to_value(source_config) + .map_err(|_| invalid("Docker workload configuration cannot be retained"))?, + ) + .map_err(|_| invalid("Docker workload configuration cannot be retained"))?; + body.host_config.clone_from(&source.host_config); + body.image = Some(image_id.clone()); + // Let the new image supply PATH and other image defaults. Only the + // driver-owned bootstrap environment belongs to the container spec. + body.env = Some( + source_config + .env + .as_deref() + .unwrap_or_default() + .iter() + .filter(|entry| { + entry.split_once('=').is_some_and(|(name, _)| { + matches!( + name, + openshell_core::sandbox_env::LOG_LEVEL + | openshell_core::sandbox_env::TELEMETRY_ENABLED + ) + }) + }) + .cloned() + .collect(), + ); + let labels = body + .labels + .as_mut() + .ok_or_else(|| invalid("Docker workload labels are missing"))?; + labels.insert(OPERATION_LABEL.into(), p.operation_id.clone()); + labels.insert(REVISION_LABEL.into(), p.revision.to_string()); + let container_name = source + .name + .as_deref() + .and_then(|name| name.strip_prefix('/')) + .filter(|name| !name.is_empty()) + .ok_or_else(|| invalid("Docker workload canonical name is missing"))? + .to_string(); + let original_bootstrap = std::fs::read(directory.join(BOUNDARY_CONFIG_FILE)) + .map_err(|_| invalid("original Docker bootstrap is unavailable"))?; + let intent = Intent { + version: 1, + namespace: self.config.sandbox_namespace.clone(), + sandbox_id: p.sandbox_id.clone(), + sandbox_name: p.name.clone(), + operation_id: p.operation_id.clone(), + revision: p.revision, + previous_image: p.expected_image.clone(), + image: p.image.clone(), + image_id, + original_container: source_id.clone(), + container_name, + target_container: None, + body, + original_bootstrap, + original_descriptor: descriptor_bytes, + phase: Phase::Prepared, + }; + // Validate both halves before accepting any destructive Docker action. + let _ = decode_boundary(&intent, "validation-only-target")?; + Ok(intent) + } + + async fn validate_update_resource( + &self, + intent: &Intent, + p: &SandboxUpdatePreconditions, + ) -> Result<(), Status> { + let id = intent + .target_container + .as_ref() + .unwrap_or(&intent.original_container); + let container = self + .docker + .inspect_container(id, None) + .await + .map_err(|_| invalid("recorded Docker update resource is unavailable"))?; + require_owner(&container, p, &self.config)?; + self.admit_container_resources(&container).await?; + if intent.phase != Phase::Prepared { + verify_replacement(&container, intent)?; + } + if intent.phase == Phase::Applied { + verify_published_boundary(intent, &self.config)?; + } + Ok(()) + } + + async fn replace_image_container( + &self, + intent: &mut Intent, + p: &SandboxUpdatePreconditions, + ) -> Result<(), Status> { + if intent.phase == Phase::Prepared { + match self + .docker + .inspect_container(&intent.original_container, None) + .await + { + Ok(source) => { + require_owner(&source, p, &self.config)?; + self.admit_container_resources(&source).await?; + if source.image.as_ref() + != serde_json::from_slice::( + &intent.original_descriptor, + ) + .map_err(|_| invalid("original Docker descriptor changed"))? + .resource_claims + .get("docker.image_identity") + { + return Err(invalid( + "original Docker image changed after update acceptance", + )); + } + // No force and no volume-removal flag. The gateway owns the + // stopped lifecycle and user data remains in its volume. + self.docker + .remove_container(&intent.original_container, None) + .await + .map_err(|_| { + Status::unavailable( + "remove stopped Docker workload; retry the accepted update", + ) + })?; + } + Err(error) if is_not_found_error(&error) => {} + Err(_) => { + return Err(Status::unavailable( + "inspect original Docker workload during update", + )); + } + } + let target = match self + .docker + .inspect_container(&intent.container_name, None) + .await + { + Ok(target) => target, + Err(error) if is_not_found_error(&error) => { + let created = self + .docker + .create_container( + Some( + CreateContainerOptionsBuilder::default() + .name(&intent.container_name) + .build(), + ), + intent.body.clone(), + ) + .await + .map_err(|_| { + Status::unavailable( + "create corrected Docker workload; retry the accepted update", + ) + })?; + let target = self + .docker + .inspect_container(&created.id, None) + .await + .map_err(|_| Status::unavailable("inspect corrected Docker workload"))?; + if target.id.as_ref() != Some(&created.id) { + return Err(invalid( + "Docker replacement inspection returned another container", + )); + } + target + } + Err(_) => return Err(Status::unavailable("inspect pending Docker replacement")), + }; + require_owner(&target, p, &self.config)?; + self.admit_container_resources(&target).await?; + verify_replacement(&target, intent)?; + let labels = target + .config + .as_ref() + .and_then(|value| value.labels.as_ref()) + .ok_or_else(|| invalid("replacement ownership is missing"))?; + if target.image.as_ref() != Some(&intent.image_id) + || labels.get(OPERATION_LABEL) != Some(&intent.operation_id) + || labels.get(REVISION_LABEL) != Some(&intent.revision.to_string()) + || target.state.as_ref().and_then(|value| value.status) + != Some(ContainerStateStatusEnum::CREATED) + { + return Err(invalid( + "Docker replacement does not match the accepted operation", + )); + } + intent.target_container = + Some(target.id.ok_or_else(|| { + invalid("Docker replacement lacks its immutable container ID") + })?); + intent.phase = Phase::Replaced; + save(&self.config, intent)?; + } + self.validate_update_resource(intent, p).await?; + let target = intent + .target_container + .as_ref() + .ok_or_else(|| invalid("Docker replacement checkpoint lacks its target"))?; + let (bootstrap, descriptor) = decode_boundary(intent, target)?; + let directory = docker_boundary_state_dir_by_id(&intent.sandbox_id, &self.config)?; + durable_write(&directory.join(BOUNDARY_CONFIG_FILE), &bootstrap)?; + durable_write(&directory.join(RUNTIME_DESCRIPTOR_FILE), &descriptor)?; + verify_published_boundary(intent, &self.config)?; + intent.phase = Phase::Applied; + save(&self.config, intent) + } +} + +/// Same weak per-resource lock pattern as the Kubernetes driver. This is a +/// driver mutex, not another lifecycle controller or endpoint service. +#[derive(Default)] +pub struct Gates { + values: std::sync::Mutex>>>, +} + +impl Gates { + pub fn gate_for(&self, id: &str) -> Arc> { + let mut values = self + .values + .lock() + .expect("Docker image-update gate poisoned"); + values.retain(|_, value| value.strong_count() > 0); + if let Some(value) = values.get(id).and_then(std::sync::Weak::upgrade) { + return value; + } + let gate = Arc::new(Mutex::new(())); + values.insert(id.into(), Arc::downgrade(&gate)); + gate + } +} + +fn record_ids(config: &DockerDriverRuntimeConfig) -> Result, Status> { + let sample = docker_boundary_state_dir_by_id("discovery", config)?; + let root = sample + .parent() + .ok_or_else(|| invalid("Docker native state root is missing"))?; + let entries = match std::fs::read_dir(root) { + Ok(entries) => entries, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(_) => return Err(invalid("Docker pending update inventory cannot be read")), + }; + let mut ids = Vec::new(); + for entry in entries { + let entry = entry.map_err(|_| invalid("Docker pending update inventory is incomplete"))?; + if !entry + .file_type() + .map_err(|_| invalid("Docker native state type is unavailable"))? + .is_dir() + { + continue; + } + let Some(id) = entry.file_name().to_str().map(str::to_owned) else { + continue; + }; + if valid_native_id(&id) + && read(config, &id)?.is_some_and(|intent| intent.phase != Phase::Applied) + { + ids.push(id); + } + } + Ok(ids) +} + +fn valid_native_id(id: &str) -> bool { + !id.is_empty() + && id.len() <= 128 + && id + .bytes() + .all(|v| v.is_ascii_alphanumeric() || matches!(v, b'-' | b'_')) +} + +fn snapshot(intent: &Intent) -> DriverSandbox { + pending_sandbox_snapshot( + &DriverSandbox { + id: intent.sandbox_id.clone(), + name: intent.sandbox_name.clone(), + namespace: intent.namespace.clone(), + workspace: intent + .body + .labels + .as_ref() + .and_then(|labels| labels.get(LABEL_SANDBOX_WORKSPACE)) + .cloned() + .unwrap_or_default(), + ..Default::default() + }, + &intent.namespace, + DriverCondition { + r#type: "Ready".into(), + status: "False".into(), + reason: openshell_core::driver_utils::CONDITION_STOPPED.into(), + message: "Stopped; an accepted workload image update awaits recovery".into(), + transition_time: None, + }, + false, + ) +} + +pub fn pending_ids(config: &DockerDriverRuntimeConfig) -> Result, Status> { + record_ids(config) +} + +pub fn pending_snapshots(config: &DockerDriverRuntimeConfig) -> Result, Status> { + record_ids(config)? + .iter() + .map(|id| { + read(config, id)? + .map(|intent| snapshot(&intent)) + .ok_or_else(|| invalid("Docker pending update changed during observation")) + }) + .collect() +} + +pub fn pending_snapshot( + config: &DockerDriverRuntimeConfig, + id: &str, + name: &str, +) -> Result, Status> { + if !id.is_empty() { + if !valid_native_id(id) { + return Ok(None); + } + let Some(intent) = read(config, id)? else { + return Ok(None); + }; + if !name.is_empty() && name != intent.sandbox_name { + return Err(invalid( + "Docker update name does not match its native identity", + )); + } + return Ok((intent.phase != Phase::Applied).then(|| snapshot(&intent))); + } + let matches = pending_snapshots(config)? + .into_iter() + .filter(|value| value.name == name) + .collect::>(); + match matches.len() { + 0 => Ok(None), + 1 => Ok(matches.into_iter().next()), + _ => Err(invalid( + "Docker pending update name is ambiguous; use its native ID", + )), + } +} + +pub fn require_settled(config: &DockerDriverRuntimeConfig, id: &str) -> Result<(), Status> { + if valid_native_id(id) && read(config, id)?.is_some_and(|intent| intent.phase != Phase::Applied) + { + return Err(invalid( + "resume the accepted Docker image update before starting the workload", + )); + } + Ok(()) +} + +pub enum LaunchValidation { + Complete, + BeforeStoppedRotation, +} + +pub fn validate_committed_resource( + config: &DockerDriverRuntimeConfig, + id: &str, + container: &ContainerInspectResponse, + validation: LaunchValidation, +) -> Result<(), Status> { + if !valid_native_id(id) { + return Ok(()); + } + if let Some(intent) = read(config, id)? { + if intent.phase != Phase::Applied { + return Err(invalid("Docker image update is not settled")); + } + verify_replacement(container, &intent)?; + match validation { + LaunchValidation::Complete => return verify_published_boundary(&intent, config), + // Use the fresh inspect, not just the earlier list response, to + // prove that credentials are rotated only on a stopped workload. + LaunchValidation::BeforeStoppedRotation => require_stopped(container)?, + } + let (bootstrap, descriptor) = read_published_boundary(&intent, config)?; + verify_boundary_resource_bindings(&intent, &bootstrap, &descriptor)?; + } + Ok(()) +} + +/// Enforce launch-pair consistency before using credentials or starting any +/// process. Resource bindings must also be checked before credential rotation. +pub fn validate_committed_boundary( + config: &DockerDriverRuntimeConfig, + id: &str, +) -> Result<(), Status> { + if valid_native_id(id) + && let Some(intent) = read(config, id)? + { + if intent.phase != Phase::Applied { + return Err(invalid("Docker image update is not settled")); + } + verify_published_boundary(&intent, config)?; + } + Ok(()) +} + +#[cfg(test)] +mod tests; diff --git a/crates/openshell-driver-docker/src/image_update/tests.rs b/crates/openshell-driver-docker/src/image_update/tests.rs new file mode 100644 index 0000000000..3ba13f7e21 --- /dev/null +++ b/crates/openshell-driver-docker/src/image_update/tests.rs @@ -0,0 +1,1212 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use super::*; +use crate::tests::{ + runtime_config, test_driver_with_config, test_launch_authentication, test_sandbox, + test_workload_identity, +}; +use crate::workspace::tests::{docker_api, fixture_for}; +use bollard::models::Volume; + +mod live; + +type Reply = (&'static str, String, u16, serde_json::Value); + +struct Fixture { + source: ContainerInspectResponse, + volume: Volume, + config: DockerDriverRuntimeConfig, + request: UpdateSandboxConfigurationRequest, + directory: PathBuf, +} + +impl Fixture { + fn new() -> Self { + let unique = openshell_core::SandboxSessionId::new().to_string(); + let mut sandbox = test_sandbox(); + sandbox.id = unique.clone(); + sandbox.workspace = "synthetic-update".into(); + let mut config = runtime_config(); + config.sandbox_namespace = format!("image-update-test-{unique}"); + config.workspace_storage = workspace::Storage::Persistent; + let (mut source, volume, config) = fixture_for(sandbox.clone(), config); + let labels = source.config.as_mut().unwrap().labels.as_mut().unwrap(); + labels.insert(LABEL_SANDBOX_NAME.into(), sandbox.name.clone()); + labels.insert(LABEL_MANAGED_BY.into(), LABEL_MANAGED_BY_VALUE.into()); + labels.insert( + LABEL_ISOLATION_ROLE.into(), + LABEL_ISOLATION_ROLE_SANDBOX.into(), + ); + labels.insert( + LABEL_ISOLATION_BACKEND.into(), + LABEL_ISOLATION_BACKEND_OPEN_SHELL.into(), + ); + source.id = Some("container-A".into()); + source.name = Some(format!("/openshell-fixture-{unique}")); + source.image = Some("sha256:immutable".into()); + let original_config = source.config.as_mut().unwrap(); + original_config.image = Some("sha256:immutable".into()); + original_config.working_dir = Some("/".into()); + original_config.entrypoint = Some(vec![SANDBOX_BINARY_PATH.into()]); + original_config.cmd = Some(vec![ + "--bootstrap".into(), + BOUNDARY_CONFIG_MOUNT_PATH.into(), + ]); + original_config.env = Some(vec![ + format!("{}=debug", openshell_core::sandbox_env::LOG_LEVEL), + format!("{}=false", openshell_core::sandbox_env::TELEMETRY_ENABLED), + "PATH=/original-image/path".into(), + ]); + source.host_config = Some(serde_json::from_value(serde_json::json!({ + "NetworkMode":"none", "CapDrop":["ALL"], "SecurityOpt":["no-new-privileges:true"], + "GroupAdd":["1236"], "Mounts":[{"Type":"volume", "Source":volume.name, "Target":"/sandbox", "ReadOnly":false, "VolumeOptions":{"NoCopy":true}}] + })).unwrap()); + source.state = Some(serde_json::from_value(serde_json::json!({"Status":"exited", "Running":false, "Paused":false, "Restarting":false})).unwrap()); + source.network_settings = + Some(serde_json::from_value(serde_json::json!({"Networks":{}})).unwrap()); + let authentication = + decode_docker_launch_authentication(&test_launch_authentication()).unwrap(); + let tls = generate_sandbox_tls_material(authentication.supervisor.session_id).unwrap(); + let mut provisioned = isolation::DockerBoundarySpec { + boundary_id: sandbox.id.clone(), + generation: authentication.supervisor.runtime_generation.to_string(), + session_id: authentication.supervisor.session_id, + session_rotation: authentication.supervisor.session_rotation, + auth_epoch: authentication.supervisor.auth_epoch, + gateway_id: authentication.gateway_id, + verification_keys: gateway_verification_keys(&authentication.verification_keys) + .unwrap(), + container_id: "container-A".into(), + image_identity: "sha256:immutable".into(), + gpu_requested: false, + listener_socket: BOUNDARY_SOCKET_MOUNT_PATH.into(), + control_socket: BOUNDARY_SOCKET_MOUNT_PATH.into(), + sandbox_tls: SandboxTlsServerConfig { + certificate_chain_path: BOUNDARY_CERTIFICATE_MOUNT_PATH.into(), + private_key_path: BOUNDARY_PRIVATE_KEY_MOUNT_PATH.into(), + }, + supervisor_tls: SandboxTlsClientConfig { + server_name: tls.server_name, + trust_anchor_pem: tls.trust_anchor_pem, + }, + host_gateway_ip: None, + workload_identity: test_workload_identity(), + child_env: HashMap::new(), + } + .provision() + .unwrap(); + let directory = prepare_docker_boundary_state_dir(&sandbox, &config).unwrap(); + let record = + workspace::record_bytes(&sandbox.id, &config, &volume, &test_workload_identity()) + .unwrap(); + durable_write(&directory.join(workspace::RECORD_FILE), &record).unwrap(); + let claim = String::from_utf8(record).unwrap(); + for claims in [ + &mut provisioned.boundary_config.resource_claims, + &mut provisioned.runtime_descriptor.resource_claims, + ] { + claims.insert(workspace::RECORD_CLAIM.into(), claim.clone()); + } + durable_write( + &directory.join(BOUNDARY_CONFIG_FILE), + &provisioned.boundary_config.encode().unwrap(), + ) + .unwrap(); + durable_write( + &directory.join(RUNTIME_DESCRIPTOR_FILE), + &serde_json::to_vec(&provisioned.runtime_descriptor).unwrap(), + ) + .unwrap(); + Self { + source, + volume, + config, + directory, + request: UpdateSandboxConfigurationRequest { + preconditions: Some(SandboxUpdatePreconditions { + sandbox_id: sandbox.id, + name: sandbox.name, + image: "corrected-image".into(), + expected_image: "old-image".into(), + expected_runtime_identity: String::new(), + operation_id: openshell_core::SandboxSessionId::new().to_string(), + validate_only: false, + revision: 1, + }), + ..Default::default() + }, + } + } + + fn p(&self) -> &SandboxUpdatePreconditions { + self.request.preconditions.as_ref().unwrap() + } + fn volume_reply(&self) -> Reply { + ( + "GET", + format!("/volumes/{}", self.volume.name), + 200, + serde_json::to_value(&self.volume).unwrap(), + ) + } + fn source_reply(&self) -> Reply { + ( + "GET", + "/containers/container-A/json".into(), + 200, + serde_json::to_value(&self.source).unwrap(), + ) + } + fn preflight(&self) -> Vec { + vec![ + self.source_reply(), + self.volume_reply(), + ( + "GET", + "/images/old-image/json".into(), + 200, + serde_json::json!({"Id":"sha256:immutable"}), + ), + ( + "GET", + "/images/corrected-image/json".into(), + 200, + serde_json::json!({"Id":"sha256:corrected", "Config":{"WorkingDir":"/sandbox", "User":"1234:1235"}}), + ), + ] + } + fn start_replies(&self, running: bool) -> Vec { + let mut target = self.target(); + let state = if running { "running" } else { "exited" }; + target.state.as_mut().unwrap().running = Some(running); + target.state.as_mut().unwrap().status = Some(if running { + ContainerStateStatusEnum::RUNNING + } else { + ContainerStateStatusEnum::EXITED + }); + vec![ + ( + "GET", + String::new(), + 200, + serde_json::json!([{ + "Id":"container-B", "Names":[target.name], "State":state, + "Labels":target.config.as_ref().unwrap().labels + }]), + ), + ( + "GET", + "/containers/container-B/json".into(), + 200, + serde_json::to_value(target).unwrap(), + ), + self.volume_reply(), + ] + } + fn target(&self) -> ContainerInspectResponse { + let mut target = self.source.clone(); + target.id = Some("container-B".into()); + target.image = Some("sha256:corrected".into()); + let config = target.config.as_mut().unwrap(); + config.image = Some("sha256:corrected".into()); + config + .env + .as_mut() + .unwrap() + .retain(|value| !value.starts_with("PATH=")); + config + .env + .as_mut() + .unwrap() + .push("PATH=/corrected-image/path".into()); + config + .labels + .as_mut() + .unwrap() + .insert(OPERATION_LABEL.into(), self.p().operation_id.clone()); + config + .labels + .as_mut() + .unwrap() + .insert(REVISION_LABEL.into(), self.p().revision.to_string()); + target.state.as_mut().unwrap().status = Some(ContainerStateStatusEnum::CREATED); + target + } + fn target_reply(&self, lookup: &str) -> Reply { + ( + "GET", + format!("/containers/{lookup}/json"), + 200, + serde_json::to_value(self.target()).unwrap(), + ) + } + fn name(&self) -> &str { + self.source + .name + .as_deref() + .unwrap() + .strip_prefix('/') + .unwrap() + } + fn after_create(&self) -> Vec { + vec![ + self.target_reply("container-B"), + self.volume_reply(), + self.target_reply("container-B"), + self.volume_reply(), + ] + } + fn create_reply(&self, status: u16) -> Reply { + ( + "POST", + format!("/containers/create?name={}&platform=", self.name()), + status, + if status == 201 { + serde_json::json!({"Id":"container-B", "Warnings":[]}) + } else { + serde_json::json!({"message":"synthetic creation interruption"}) + }, + ) + } + fn missing(id: &str) -> Reply { + ( + "GET", + format!("/containers/{id}/json"), + 404, + serde_json::json!({"message":"not found"}), + ) + } + fn apply_replies(&self) -> Vec { + let mut replies = self.preflight(); + replies.extend([ + self.source_reply(), + self.volume_reply(), + ( + "DELETE", + "/containers/container-A".into(), + 204, + serde_json::Value::Null, + ), + Self::missing(self.name()), + self.create_reply(201), + ]); + replies.extend(self.after_create()); + replies + } + async fn call( + &self, + replies: Vec, + request: &UpdateSandboxConfigurationRequest, + ) -> (Result<(), Status>, Vec) { + let (docker, server) = docker_api(replies).await; + let mut driver = test_driver_with_config(self.config.clone()); + driver.docker = Arc::new(docker); + let result = Box::pin(driver.update_workload_image(request)).await; + (result, server.await.unwrap()) + } + + async fn grpc_call( + &self, + config: DockerDriverRuntimeConfig, + replies: Vec, + request: UpdateSandboxConfigurationRequest, + ) -> Result<(), Status> { + use openshell_core::proto::compute::v1::{ + compute_driver_configuration_updates_client::ComputeDriverConfigurationUpdatesClient, + compute_driver_configuration_updates_server::ComputeDriverConfigurationUpdatesServer, + }; + let (docker, api) = docker_api(replies).await; + let mut driver = test_driver_with_config(config); + driver.docker = Arc::new(docker); + let service = ComputeDriverService::new_in_process(driver); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let (shutdown, stopped) = oneshot::channel(); + let server = tokio::spawn(async move { + tonic::transport::Server::builder() + .add_service(ComputeDriverConfigurationUpdatesServer::new(service)) + .serve_with_incoming_shutdown( + tokio_stream::wrappers::TcpListenerStream::new(listener), + async { + let _ = stopped.await; + }, + ) + .await + .unwrap(); + }); + let mut client = + ComputeDriverConfigurationUpdatesClient::connect(format!("http://{address}")) + .await + .unwrap(); + let result = client + .update_sandbox_configuration(request) + .await + .map(|_| ()); + let _ = shutdown.send(()); + server.await.unwrap(); + api.await.unwrap(); + result + } +} + +impl Drop for Fixture { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.directory); + if let Some(parent) = self.directory.parent() { + let _ = std::fs::remove_dir(parent); + } + } +} + +#[tokio::test] +async fn persistent_update_requires_the_private_workspace_record() { + for mutation in ["missing", "invalid", "foreign-namespace", "foreign-uid"] { + let fixture = Fixture::new(); + let path = fixture.directory.join(workspace::RECORD_FILE); + match mutation { + "missing" => std::fs::remove_file(&path).unwrap(), + "invalid" => std::fs::write(&path, "not-json").unwrap(), + _ => { + let mut record: serde_json::Value = + serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); + if mutation == "foreign-namespace" { + record["namespace"] = "foreign".into(); + } else { + record["uid"] = 2000.into(); + } + std::fs::write(&path, serde_json::to_vec(&record).unwrap()).unwrap(); + } + } + let (result, requests) = fixture + .call(vec![fixture.source_reply()], &fixture.request) + .await; + assert_eq!(result.unwrap_err().code(), tonic::Code::FailedPrecondition); + assert_eq!(requests.len(), 1); + assert!( + read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .is_none() + ); + } +} + +#[tokio::test] +async fn replay_requires_storage_authority_before_any_docker_action() { + for mutation in ["missing", "changed"] { + let fixture = Fixture::new(); + let (docker, server) = docker_api(fixture.preflight()).await; + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = Arc::new(docker); + let intent = driver.prepare_image_update(fixture.p()).await.unwrap(); + server.await.unwrap(); + save(&fixture.config, &intent).unwrap(); + let checkpoint = + std::fs::read(path(&fixture.config, &fixture.p().sandbox_id).unwrap()).unwrap(); + let record = fixture.directory.join(workspace::RECORD_FILE); + if mutation == "missing" { + std::fs::remove_file(&record).unwrap(); + } else { + let mut value: serde_json::Value = + serde_json::from_slice(&std::fs::read(&record).unwrap()).unwrap(); + value["uid"] = 2000.into(); + std::fs::write(&record, serde_json::to_vec(&value).unwrap()).unwrap(); + } + let (result, requests) = fixture.call(vec![], &fixture.request).await; + assert_eq!(result.unwrap_err().code(), tonic::Code::FailedPrecondition); + assert!(requests.is_empty()); + assert_eq!( + std::fs::read(path(&fixture.config, &fixture.p().sandbox_id).unwrap()).unwrap(), + checkpoint + ); + } +} + +#[tokio::test] +async fn incompatible_target_identity_is_rejected_before_checkpoint_or_deletion() { + // Symbolic and implicit identities require account lookup. The initial + // updater rejects those representations rather than assuming that the + // source identity can execute a different image's application. + for user in [ + "2000:2000", + "1234:2000", + "other:other", + "1234:other", + "1234", + "", + ] { + let fixture = Fixture::new(); + let bootstrap = std::fs::read(fixture.directory.join(BOUNDARY_CONFIG_FILE)).unwrap(); + let descriptor = std::fs::read(fixture.directory.join(RUNTIME_DESCRIPTOR_FILE)).unwrap(); + let mut replies = fixture.preflight(); + replies.last_mut().unwrap().3["Config"]["User"] = user.into(); + let (result, requests) = fixture.call(replies, &fixture.request).await; + assert!( + matches!(result, Err(ref error) if error.code() == tonic::Code::FailedPrecondition), + "accepted target USER {user:?}" + ); + // The fake endpoint accepts only the four read-only preflight calls. + assert_eq!(requests.len(), 4); + assert!( + read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .is_none() + ); + assert_eq!( + std::fs::read(fixture.directory.join(BOUNDARY_CONFIG_FILE)).unwrap(), + bootstrap + ); + assert_eq!( + std::fs::read(fixture.directory.join(RUNTIME_DESCRIPTOR_FILE)).unwrap(), + descriptor + ); + } +} + +#[tokio::test] +async fn validation_only_checks_original_without_writing_an_intent_or_replacing() { + let fixture = Fixture::new(); + let mut request = fixture.request.clone(); + request.preconditions.as_mut().unwrap().validate_only = true; + let original = std::fs::read(fixture.directory.join(RUNTIME_DESCRIPTOR_FILE)).unwrap(); + let mut replies = fixture.preflight(); + replies.extend([fixture.source_reply(), fixture.volume_reply()]); + fixture.call(replies, &request).await.0.unwrap(); + assert!( + read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .is_none() + ); + assert_eq!( + std::fs::read(fixture.directory.join(RUNTIME_DESCRIPTOR_FILE)).unwrap(), + original + ); +} + +#[tokio::test] +async fn image_replacement_keeps_native_and_volume_identity_and_publishes_new_evidence() { + let fixture = Fixture::new(); + let (result, requests) = fixture + .call(fixture.apply_replies(), &fixture.request) + .await; + result.unwrap(); + let intent = read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .unwrap(); + assert!(intent.phase == Phase::Applied); + assert_eq!(intent.sandbox_id, fixture.p().sandbox_id); + assert_eq!(intent.target_container.as_deref(), Some("container-B")); + assert_eq!(requests[8]["Image"], "sha256:corrected"); + assert!( + !requests[8]["Env"] + .as_array() + .unwrap() + .iter() + .any(|value| value.as_str().unwrap().starts_with("PATH=")) + ); + let descriptor: SandboxRuntimeDescriptor = serde_json::from_slice( + &std::fs::read(fixture.directory.join(RUNTIME_DESCRIPTOR_FILE)).unwrap(), + ) + .unwrap(); + assert_eq!( + descriptor.resource_claims["docker.container_id"], + "container-B" + ); + assert_eq!( + descriptor.resource_claims["docker.image_identity"], + "sha256:corrected" + ); + assert_eq!( + descriptor.resource_claims[workspace::RECORD_CLAIM], + workspace::record_claim(&fixture.p().sandbox_id, &fixture.config) + .unwrap() + .unwrap() + ); +} + +#[tokio::test] +async fn running_original_refuses_before_volume_read_or_checkpoint() { + let mut fixture = Fixture::new(); + fixture.source.state.as_mut().unwrap().running = Some(true); + fixture.source.state.as_mut().unwrap().status = Some(ContainerStateStatusEnum::RUNNING); + let error = fixture + .call(vec![fixture.source_reply()], &fixture.request) + .await + .0 + .unwrap_err(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert!( + read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .is_none() + ); +} + +#[tokio::test] +async fn failed_target_creation_recovers_same_operation_in_a_fresh_driver() { + let fixture = Fixture::new(); + let mut replies = fixture.preflight(); + replies.extend([ + fixture.source_reply(), + fixture.volume_reply(), + ( + "DELETE", + "/containers/container-A".into(), + 204, + serde_json::Value::Null, + ), + Fixture::missing(fixture.name()), + fixture.create_reply(500), + ]); + assert_eq!( + fixture + .call(replies, &fixture.request) + .await + .0 + .unwrap_err() + .code(), + tonic::Code::Unavailable + ); + let before = read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .unwrap(); + assert!(before.phase == Phase::Prepared); + let mut recovery = vec![ + Fixture::missing("container-A"), + Fixture::missing(fixture.name()), + fixture.create_reply(201), + ]; + recovery.extend(fixture.after_create()); + fixture.call(recovery, &fixture.request).await.0.unwrap(); + let applied = read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .unwrap(); + assert!(applied.phase == Phase::Applied); + assert_eq!(applied.operation_id, before.operation_id); + fixture + .call( + vec![fixture.target_reply("container-B"), fixture.volume_reply()], + &fixture.request, + ) + .await + .0 + .unwrap(); +} + +#[tokio::test] +async fn target_created_before_checkpoint_is_reconciled_without_another_create() { + let fixture = Fixture::new(); + let (docker, server) = docker_api(fixture.preflight()).await; + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = Arc::new(docker); + let intent = driver.prepare_image_update(fixture.p()).await.unwrap(); + server.await.unwrap(); + save(&fixture.config, &intent).unwrap(); + let mut replies = vec![ + Fixture::missing("container-A"), + fixture.target_reply(fixture.name()), + fixture.volume_reply(), + ]; + replies.extend([fixture.target_reply("container-B"), fixture.volume_reply()]); + fixture.call(replies, &fixture.request).await.0.unwrap(); + assert!( + read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .unwrap() + .phase + == Phase::Applied + ); +} + +#[test] +fn request_validation_rejects_paths_unsupported_mutations_and_foreign_compute_identity() { + let fixture = Fixture::new(); + let mut request = fixture.request.clone(); + request.preconditions.as_mut().unwrap().sandbox_id = "../foreign".into(); + assert!(validate_request(&request).is_err()); + request = fixture.request.clone(); + request + .preconditions + .as_mut() + .unwrap() + .expected_runtime_identity = "foreign".into(); + assert!(validate_request(&request).is_err()); + request = fixture.request.clone(); + request.environment = + Some(openshell_core::proto::compute::v1::SandboxEnvironmentUpdate::default()); + assert_eq!( + validate_request(&request).unwrap_err().code(), + tonic::Code::Unimplemented + ); +} + +#[tokio::test] +async fn interrupted_boundary_publication_is_repaired_from_the_same_native_intent() { + let fixture = Fixture::new(); + fixture + .call(fixture.apply_replies(), &fixture.request) + .await + .0 + .unwrap(); + let mut intent = read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .unwrap(); + intent.phase = Phase::Replaced; + save(&fixture.config, &intent).unwrap(); + durable_write( + &fixture.directory.join(BOUNDARY_CONFIG_FILE), + b"{incomplete", + ) + .unwrap(); + fixture + .call( + vec![fixture.target_reply("container-B"), fixture.volume_reply()], + &fixture.request, + ) + .await + .0 + .unwrap(); + let applied = read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .unwrap(); + assert!(applied.phase == Phase::Applied); + verify_published_boundary(&applied, &fixture.config).unwrap(); + assert_eq!(applied.operation_id, intent.operation_id); +} + +#[tokio::test] +async fn same_name_target_with_another_operation_is_not_adopted() { + let fixture = Fixture::new(); + let (docker, server) = docker_api(fixture.preflight()).await; + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = Arc::new(docker); + let intent = driver.prepare_image_update(fixture.p()).await.unwrap(); + server.await.unwrap(); + save(&fixture.config, &intent).unwrap(); + let mut foreign = fixture.target(); + foreign + .config + .as_mut() + .unwrap() + .labels + .as_mut() + .unwrap() + .insert(OPERATION_LABEL.into(), "foreign-operation".into()); + let replies = vec![ + Fixture::missing("container-A"), + ( + "GET", + format!("/containers/{}/json", fixture.name()), + 200, + serde_json::to_value(foreign).unwrap(), + ), + fixture.volume_reply(), + ]; + let error = fixture.call(replies, &fixture.request).await.0.unwrap_err(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert!( + read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .unwrap() + .phase + == Phase::Prepared + ); + assert_eq!( + std::fs::read(fixture.directory.join(RUNTIME_DESCRIPTOR_FILE)).unwrap(), + intent.original_descriptor + ); +} + +#[tokio::test] +async fn conflicting_replay_refuses_before_docker_or_checkpoint_writes() { + let fixture = Fixture::new(); + fixture + .call(fixture.apply_replies(), &fixture.request) + .await + .0 + .unwrap(); + let checkpoint = + std::fs::read(path(&fixture.config, &fixture.p().sandbox_id).unwrap()).unwrap(); + let mut conflicting = fixture.request.clone(); + conflicting.preconditions.as_mut().unwrap().operation_id = + openshell_core::SandboxSessionId::new().to_string(); + let error = fixture.call(vec![], &conflicting).await.0.unwrap_err(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert_eq!( + std::fs::read(path(&fixture.config, &fixture.p().sandbox_id).unwrap()).unwrap(), + checkpoint + ); +} + +#[tokio::test] +async fn replacement_cannot_substitute_another_persistent_volume_record() { + let fixture = Fixture::new(); + let (docker, server) = docker_api(fixture.preflight()).await; + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = Arc::new(docker); + let intent = driver.prepare_image_update(fixture.p()).await.unwrap(); + server.await.unwrap(); + let mut target = fixture.target(); + verify_replacement(&target, &intent).unwrap(); + target + .host_config + .as_mut() + .unwrap() + .mounts + .as_mut() + .unwrap()[0] + .source = Some("foreign-volume".into()); + assert_eq!( + verify_replacement(&target, &intent).unwrap_err().code(), + tonic::Code::FailedPrecondition + ); +} + +#[tokio::test] +async fn pending_update_survives_driver_restart_and_blocks_start_by_id_or_name() { + let fixture = Fixture::new(); + let (docker, server) = docker_api(fixture.preflight()).await; + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = Arc::new(docker); + let intent = driver.prepare_image_update(fixture.p()).await.unwrap(); + server.await.unwrap(); + save(&fixture.config, &intent).unwrap(); + let snapshots = pending_snapshots(&fixture.config).unwrap(); + assert_eq!(snapshots.len(), 1); + assert_eq!(snapshots[0].id, fixture.p().sandbox_id); + assert_eq!( + snapshots[0].status.as_ref().unwrap().conditions[0].reason, + openshell_core::driver_utils::CONDITION_STOPPED + ); + assert_eq!( + pending_ids(&fixture.config).unwrap(), + vec![fixture.p().sandbox_id.clone()] + ); + let (docker, server) = docker_api(vec![]).await; + let mut reopened = test_driver_with_config(fixture.config.clone()); + reopened.docker = Arc::new(docker); + let observed = reopened + .get_sandbox_snapshot("", &fixture.p().name) + .await + .unwrap() + .unwrap(); + assert_eq!(observed.id, fixture.p().sandbox_id); + for id in [fixture.p().sandbox_id.as_str(), ""] { + let error = Box::pin(reopened.start_sandbox( + id, + &fixture.p().name, + "generation-2", + &test_launch_authentication(), + None, + )) + .await + .unwrap_err(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + } + server.await.unwrap(); + assert!( + read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .unwrap() + .phase + == Phase::Prepared + ); +} + +#[tokio::test] +async fn completed_update_clears_pending_inventory_and_keeps_start_bound_to_target() { + let fixture = Fixture::new(); + fixture + .call(fixture.apply_replies(), &fixture.request) + .await + .0 + .unwrap(); + assert!(pending_ids(&fixture.config).unwrap().is_empty()); + assert!( + pending_snapshot(&fixture.config, &fixture.p().sandbox_id, &fixture.p().name) + .unwrap() + .is_none() + ); + require_settled(&fixture.config, &fixture.p().sandbox_id).unwrap(); + let mut target = fixture.target(); + validate_committed_resource( + &fixture.config, + &fixture.p().sandbox_id, + &target, + LaunchValidation::Complete, + ) + .unwrap(); + target.id = Some("same-name-foreign-container".into()); + assert!( + validate_committed_resource( + &fixture.config, + &fixture.p().sandbox_id, + &target, + LaunchValidation::Complete + ) + .is_err() + ); +} + +#[tokio::test] +async fn stopped_start_repairs_interrupted_launch_rotation_before_staging() { + let fixture = Fixture::new(); + fixture + .call(fixture.apply_replies(), &fixture.request) + .await + .0 + .unwrap(); + let descriptor_path = fixture.directory.join(RUNTIME_DESCRIPTOR_FILE); + let previous_descriptor = std::fs::read(&descriptor_path).unwrap(); + let mut interrupted = + decode_docker_launch_authentication(&test_launch_authentication()).unwrap(); + interrupted.supervisor.runtime_generation = + openshell_core::sandbox_generation::SandboxGenerationId::parse("generation-interrupted") + .unwrap(); + let interrupted_authentication = serde_json::to_vec(&interrupted).unwrap(); + refresh_docker_boundary_authentication( + &fixture.p().sandbox_id, + &fixture.config, + &interrupted_authentication, + ) + .await + .unwrap(); + // Model exit after bootstrap publication but before descriptor publication. + std::fs::write(&descriptor_path, previous_descriptor).unwrap(); + let intent = read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .unwrap(); + assert!(verify_published_boundary(&intent, &fixture.config).is_err()); + + let (docker, server) = docker_api(fixture.start_replies(false)).await; + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = Arc::new(docker); + let mut authentication = + decode_docker_launch_authentication(&test_launch_authentication()).unwrap(); + authentication.supervisor.runtime_generation = + openshell_core::sandbox_generation::SandboxGenerationId::parse("generation-retry").unwrap(); + let encoded = serde_json::to_vec(&authentication).unwrap(); + let error = Box::pin(driver.start_sandbox( + &fixture.p().sandbox_id, + &fixture.p().name, + "generation-retry", + &encoded, + None, + )) + .await + .unwrap_err(); + server.await.unwrap(); + // Deliberately omit the workspace-root staging input: reaching this gate + // proves the real Start path repaired publication, without starting a fake + // workload or requiring a supervisor process in this unit test. + assert!(error.message().contains("workspace for restart"), "{error}"); + verify_published_boundary(&intent, &fixture.config).unwrap(); + let repaired = read_docker_runtime_descriptor(&fixture.p().sandbox_id, &fixture.config) + .await + .unwrap() + .unwrap(); + assert_eq!(repaired.generation, "generation-retry"); + assert_eq!(repaired.session_id, authentication.supervisor.session_id); + assert_eq!( + repaired.resource_claims["docker.container_id"], + "container-B" + ); + assert_eq!( + repaired.resource_claims["docker.image_identity"], + "sha256:corrected" + ); +} + +#[tokio::test] +async fn interrupted_launch_pair_requires_stopped_workload_and_fresh_authentication() { + for (running, fresh_authentication, inspected_running) in [ + (true, true, true), + (false, false, false), + (false, true, true), + ] { + let fixture = Fixture::new(); + fixture + .call(fixture.apply_replies(), &fixture.request) + .await + .0 + .unwrap(); + let descriptor_path = fixture.directory.join(RUNTIME_DESCRIPTOR_FILE); + let previous_descriptor = std::fs::read(&descriptor_path).unwrap(); + refresh_docker_boundary_authentication( + &fixture.p().sandbox_id, + &fixture.config, + &test_launch_authentication(), + ) + .await + .unwrap(); + std::fs::write(&descriptor_path, &previous_descriptor).unwrap(); + let previous_bootstrap = + std::fs::read(fixture.directory.join(BOUNDARY_CONFIG_FILE)).unwrap(); + let mut replies = fixture.start_replies(running); + if inspected_running { + replies[1].3["State"]["Running"] = true.into(); + replies[1].3["State"]["Status"] = "running".into(); + } + let (docker, server) = docker_api(replies).await; + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = Arc::new(docker); + let authentication = if fresh_authentication { + test_launch_authentication() + } else { + Vec::new() + }; + let error = Box::pin(driver.start_sandbox( + &fixture.p().sandbox_id, + &fixture.p().name, + "generation-1", + &authentication, + None, + )) + .await + .unwrap_err(); + server.await.unwrap(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert!( + error.message().contains(if !running && inspected_running { + "verifiably stopped" + } else { + "runtime evidence is inconsistent" + }), + "{error}" + ); + assert_eq!( + std::fs::read(&descriptor_path).unwrap(), + previous_descriptor + ); + assert_eq!( + std::fs::read(fixture.directory.join(BOUNDARY_CONFIG_FILE)).unwrap(), + previous_bootstrap + ); + } +} + +#[tokio::test] +async fn launch_repair_never_adopts_changed_immutable_bindings() { + let fixture = Fixture::new(); + fixture + .call(fixture.apply_replies(), &fixture.request) + .await + .0 + .unwrap(); + for filename in [BOUNDARY_CONFIG_FILE, RUNTIME_DESCRIPTOR_FILE] { + let path = fixture.directory.join(filename); + let original = std::fs::read(&path).unwrap(); + for (pointer, foreign) in [ + ("/boundary_id", serde_json::json!("foreign-sandbox")), + ( + "/resource_claims/docker.container_id", + serde_json::json!("foreign-container"), + ), + ( + "/resource_claims/docker.image_identity", + serde_json::json!("sha256:foreign"), + ), + ("/workload_identity/uid", serde_json::json!(2000)), + ("/workload_identity/gid", serde_json::json!(2000)), + ] { + let mut changed: serde_json::Value = serde_json::from_slice(&original).unwrap(); + *changed.pointer_mut(pointer).unwrap() = foreign; + let changed = serde_json::to_vec(&changed).unwrap(); + std::fs::write(&path, &changed).unwrap(); + let mut replies = fixture.start_replies(false); + if filename == RUNTIME_DESCRIPTOR_FILE + && matches!( + pointer, + "/boundary_id" | "/workload_identity/uid" | "/workload_identity/gid" + ) + { + replies.pop(); // The protected record rejects before volume access. + } + let (docker, server) = docker_api(replies).await; + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = Arc::new(docker); + let error = Box::pin(driver.start_sandbox( + &fixture.p().sandbox_id, + &fixture.p().name, + "generation-1", + &test_launch_authentication(), + None, + )) + .await + .unwrap_err(); + server.await.unwrap(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert_eq!(std::fs::read(&path).unwrap(), changed); + } + std::fs::write(path, original).unwrap(); + } + // Native container/image/workspace substitutions must also fail before + // credential publication, independently of the private file checks. + for (pointer, foreign) in [ + ("/Id", "foreign-container"), + ("/Image", "sha256:foreign"), + ("/HostConfig/Mounts/0/Source", "foreign-workspace"), + ] { + let mut target = serde_json::to_value(fixture.target()).unwrap(); + *target.pointer_mut(pointer).unwrap() = foreign.into(); + let target: ContainerInspectResponse = serde_json::from_value(target).unwrap(); + assert!( + validate_committed_resource( + &fixture.config, + &fixture.p().sandbox_id, + &target, + LaunchValidation::BeforeStoppedRotation + ) + .is_err() + ); + } +} + +async fn prepared_fixture() -> Fixture { + let fixture = Fixture::new(); + let (docker, server) = docker_api(fixture.preflight()).await; + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = Arc::new(docker); + let intent = driver.prepare_image_update(fixture.p()).await.unwrap(); + server.await.unwrap(); + save(&fixture.config, &intent).unwrap(); + fixture +} + +#[tokio::test] +async fn startup_cleanup_and_stop_preserve_pending_update_without_a_workload() { + let fixture = prepared_fixture().await; + let mut channel = fixture.volume.clone(); + channel.name = docker_channel_volume_name_by_id(&fixture.p().sandbox_id, &fixture.config); + channel.labels.remove(workspace::ROLE_LABEL); + // All three list operations are read-only. Any removal during startup or + // Stop would fail the fixture's next expected method. + let replies = vec![ + ("GET", String::new(), 200, serde_json::json!([])), + ("GET", String::new(), 200, serde_json::json!([])), + ( + "GET", + String::new(), + 200, + serde_json::json!({"Volumes":[fixture.volume, channel]}), + ), + ("GET", String::new(), 200, serde_json::json!([])), + ("GET", String::new(), 200, serde_json::json!([])), + ]; + let (docker, server) = docker_api(replies).await; + let mut reopened = test_driver_with_config(fixture.config.clone()); + reopened.docker = Arc::new(docker); + reopened + .reconcile_runtime_resources_at_startup() + .await + .unwrap(); + ComputeDriver::stop_sandbox( + &reopened, + Request::new(StopSandboxRequest { + sandbox_id: String::new(), + name: fixture.p().name.clone(), + }), + ) + .await + .unwrap(); + server.await.unwrap(); + assert_eq!( + pending_ids(&fixture.config).unwrap(), + vec![fixture.p().sandbox_id.clone()] + ); + assert!(fixture.directory.join(RUNTIME_DESCRIPTOR_FILE).is_file()); +} + +#[tokio::test] +async fn explicit_delete_clears_pending_gap_and_reports_the_deleted_native_resource() { + let fixture = prepared_fixture().await; + let mut replies = vec![ + ("GET", String::new(), 200, serde_json::json!([])), + ("GET", String::new(), 200, serde_json::json!([])), + fixture.volume_reply(), + ( + "DELETE", + format!("/volumes/{}", fixture.volume.name), + 204, + serde_json::Value::Null, + ), + ]; + for name in [ + docker_channel_volume_name_by_id(&fixture.p().sandbox_id, &fixture.config), + docker_supervisor_volume_name_by_id(&fixture.p().sandbox_id, &fixture.config), + docker_control_volume_name_by_id(&fixture.p().sandbox_id, &fixture.config), + ] { + replies.push(( + "DELETE", + format!("/volumes/{name}"), + 204, + serde_json::Value::Null, + )); + } + let (docker, server) = docker_api(replies).await; + let mut reopened = test_driver_with_config(fixture.config.clone()); + reopened.docker = Arc::new(docker); + let response = ComputeDriver::delete_sandbox( + &reopened, + Request::new(DeleteSandboxRequest { + sandbox_id: String::new(), + name: fixture.p().name.clone(), + }), + ) + .await + .unwrap() + .into_inner(); + server.await.unwrap(); + assert!(response.deleted); + assert!(pending_ids(&fixture.config).unwrap().is_empty()); + assert!(!fixture.directory.exists()); +} + +#[tokio::test] +async fn optional_rpc_remains_disabled_without_persistent_storage() { + let fixture = Fixture::new(); + let mut config = fixture.config.clone(); + config.workspace_storage = workspace::Storage::WritableLayer; + let capability = openshell_core::extension_protocol::COMPUTE_UPDATE_SANDBOX_CONFIGURATION; + let advertised = test_driver_with_config(config.clone()) + .capabilities() + .extension + .unwrap() + .supported_capabilities; + assert!(!advertised.iter().any(|value| value == capability)); + let error = fixture + .grpc_call(config, vec![], fixture.request.clone()) + .await + .unwrap_err(); + assert_eq!(error.code(), tonic::Code::Unimplemented); + assert!( + read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .is_none() + ); +} + +#[tokio::test] +async fn optional_rpc_uses_the_existing_preflight_contract_when_enabled() { + let fixture = Fixture::new(); + let capability = openshell_core::extension_protocol::COMPUTE_UPDATE_SANDBOX_CONFIGURATION; + let advertised = test_driver_with_config(fixture.config.clone()) + .capabilities() + .extension + .unwrap() + .supported_capabilities; + assert!(advertised.iter().any(|value| value == capability)); + let mut request = fixture.request.clone(); + request.preconditions.as_mut().unwrap().validate_only = true; + let mut replies = fixture.preflight(); + replies.extend([fixture.source_reply(), fixture.volume_reply()]); + fixture + .grpc_call(fixture.config.clone(), replies, request) + .await + .unwrap(); + assert!( + read(&fixture.config, &fixture.p().sandbox_id) + .unwrap() + .is_none() + ); +} diff --git a/crates/openshell-driver-docker/src/image_update/tests/live.rs b/crates/openshell-driver-docker/src/image_update/tests/live.rs new file mode 100644 index 0000000000..2d8bb92a08 --- /dev/null +++ b/crates/openshell-driver-docker/src/image_update/tests/live.rs @@ -0,0 +1,268 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Opt-in real-Docker driver integration; not gateway/supervisor protocol E2E. +//! See the crate README for image preparation and invocation. + +use super::*; +use std::process::Command; + +struct Resources { + container: String, + volume: String, + sandbox: String, +} + +impl Drop for Resources { + fn drop(&mut self) { + // No force, broad selectors, or removal of unverified resources, even + // when an assertion fails halfway through replacement. + for (kind, name, label_path) in [ + ("container", &self.container, ".Config.Labels"), + ("volume", &self.volume, ".Labels"), + ] { + let format = format!("{{{{index {label_path} \"{LABEL_SANDBOX_ID}\"}}}}"); + let owned = Command::new("docker") + .args([kind, "inspect", "--format", &format, name]) + .output() + .is_ok_and(|output| { + output.status.success() + && String::from_utf8_lossy(&output.stdout).trim() == self.sandbox + }); + if owned { + let removed = Command::new("docker").args([kind, "rm", name]).output(); + if !removed.is_ok_and(|output| output.status.success()) { + eprintln!("live Docker test could not remove its owned {kind} {name}"); + } + } + } + } +} + +fn archived_file(docker_id: &str, path: &str) -> Vec { + let output = Command::new("docker") + .args(["cp", &format!("{docker_id}:{path}"), "-"]) + .output() + .unwrap(); + assert!(output.status.success()); + let mut archive = tar::Archive::new(output.stdout.as_slice()); + let mut bytes = Vec::new(); + std::io::Read::read_to_end( + &mut archive.entries().unwrap().next().unwrap().unwrap(), + &mut bytes, + ) + .unwrap(); + bytes +} + +#[tokio::test] +#[ignore = "requires a live Docker daemon and the four prepared test images; see README"] +async fn live_stopped_image_replacement_retains_data_and_recovers_a_deleted_source() { + let images: Vec = ["A", "B", "BAD", "SYMBOLIC"] + .into_iter() + .map(|suffix| { + std::env::var(format!("OPENSHELL_DOCKER_UPDATE_IMAGE_{suffix}")) + .expect("prepare live test images first") + }) + .collect(); + let docker = Arc::new(Docker::connect_with_local_defaults().unwrap()); + docker.ping().await.unwrap(); + let mut fixture = Fixture::new(); + let resources = Resources { + container: fixture.name().into(), + volume: fixture.volume.name.clone(), + sandbox: fixture.p().sandbox_id.clone(), + }; + assert!( + matches!(docker.inspect_volume(&resources.volume).await, Err(ref error) if is_not_found_error(error)) + ); + let volume = docker + .create_volume(VolumeCreateRequest { + name: Some(resources.volume.clone()), + driver: Some("local".into()), + labels: Some(fixture.volume.labels.clone()), + ..Default::default() + }) + .await + .unwrap(); + let mut body: ContainerCreateBody = + serde_json::from_value(serde_json::to_value(&fixture.source.config).unwrap()).unwrap(); + body.host_config = fixture.source.host_config.clone(); + body.image = Some(images[0].clone()); + let private_record = workspace::record_bytes( + &resources.sandbox, + &fixture.config, + &volume, + &test_workload_identity(), + ) + .unwrap(); + durable_write( + &fixture.directory.join(workspace::RECORD_FILE), + &private_record, + ) + .unwrap(); + let source_id = docker + .create_container( + Some( + CreateContainerOptionsBuilder::default() + .name(&resources.container) + .build(), + ), + body, + ) + .await + .unwrap() + .id; + let source = docker.inspect_container(&source_id, None).await.unwrap(); + for filename in [BOUNDARY_CONFIG_FILE, RUNTIME_DESCRIPTOR_FILE] { + let path = fixture.directory.join(filename); + let mut value: serde_json::Value = + serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); + value["resource_claims"]["docker.container_id"] = source_id.clone().into(); + value["resource_claims"]["docker.image_identity"] = source.image.clone().unwrap().into(); + value["resource_claims"][workspace::RECORD_CLAIM] = + String::from_utf8(private_record.clone()).unwrap().into(); + value["workload_identity"]["resource_digest"] = source.image.clone().unwrap().into(); + value["provisioning_evidence"] = serde_json::to_value( + isolation::provisioning_evidence(&source_id, value["generation"].as_str().unwrap()) + .unwrap(), + ) + .unwrap(); + durable_write(&path, &serde_json::to_vec(&value).unwrap()).unwrap(); + } + let mut archive = tar::Builder::new(Vec::new()); + let marker = b"retained workspace and session proof\n"; + let mut header = tar::Header::new_gnu(); + header.set_size(marker.len() as u64); + header.set_mode(0o600); + header.set_uid(1234); + header.set_gid(1235); + header.set_cksum(); + archive + .append_data(&mut header, "retained-session.txt", marker.as_slice()) + .unwrap(); + docker + .upload_to_container( + &source_id, + Some( + UploadToContainerOptionsBuilder::default() + .path("/sandbox") + .copy_uidgid("true") + .build(), + ), + bollard::body_full(Bytes::from(archive.into_inner().unwrap())), + ) + .await + .unwrap(); + let mut driver = test_driver_with_config(fixture.config.clone()); + driver.docker = docker.clone(); + fixture + .request + .preconditions + .as_mut() + .unwrap() + .expected_image = images[0].clone(); + for rejected_image in &images[2..] { + fixture.request.preconditions.as_mut().unwrap().image = rejected_image.clone(); + let error = Box::pin(driver.update_workload_image(&fixture.request)) + .await + .unwrap_err(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert!(error.message().contains("matching numeric"), "{error}"); + assert!(read(&fixture.config, &resources.sandbox).unwrap().is_none()); + assert_eq!( + docker + .inspect_container(&source_id, None) + .await + .unwrap() + .id + .as_deref(), + Some(source_id.as_str()) + ); + } + fixture.request.preconditions.as_mut().unwrap().image = images[1].clone(); + Box::pin(driver.update_workload_image(&fixture.request)) + .await + .unwrap(); + let intent = read(&fixture.config, &resources.sandbox).unwrap().unwrap(); + assert!(intent.phase == Phase::Applied); + let first_target = intent.target_container.clone().unwrap(); + assert_ne!(first_target, source_id); + let target = docker.inspect_container(&first_target, None).await.unwrap(); + assert_eq!(target.state.as_ref().unwrap().running, Some(false)); + assert_eq!( + target.config.as_ref().unwrap().user.as_deref(), + Some("1234:1235") + ); + assert_eq!( + target.config.as_ref().unwrap().labels.as_ref().unwrap()[LABEL_SANDBOX_ID], + resources.sandbox + ); + assert_eq!( + archived_file(&first_target, "/sandbox/retained-session.txt"), + marker + ); + assert_eq!(archived_file(&first_target, "/app/version"), b"B\n"); + // Reopen the driver and replay: no second replacement of the applied target. + let mut reopened = test_driver_with_config(fixture.config.clone()); + reopened.docker = docker.clone(); + Box::pin(reopened.update_workload_image(&fixture.request)) + .await + .unwrap(); + assert_eq!( + read(&fixture.config, &resources.sandbox) + .unwrap() + .unwrap() + .target_container + .as_deref(), + Some(first_target.as_str()) + ); + + // Inject exit after an accepted update has removed its stopped source. + let p = fixture.request.preconditions.as_mut().unwrap(); + p.expected_image = images[1].clone(); + p.image = images[0].clone(); + p.revision = 2; + p.operation_id = openshell_core::SandboxSessionId::new().to_string(); + let pending = reopened.prepare_image_update(fixture.p()).await.unwrap(); + save(&fixture.config, &pending).unwrap(); + docker.remove_container(&first_target, None).await.unwrap(); + assert!(require_settled(&fixture.config, &resources.sandbox).is_err()); + let mut recovered = test_driver_with_config(fixture.config.clone()); + recovered.docker = docker.clone(); + Box::pin(recovered.update_workload_image(&fixture.request)) + .await + .unwrap(); + let applied = read(&fixture.config, &resources.sandbox).unwrap().unwrap(); + let final_target = applied.target_container.as_ref().unwrap(); + assert!(applied.phase == Phase::Applied); + assert_eq!( + archived_file(final_target, "/sandbox/retained-session.txt"), + marker + ); + assert_eq!(archived_file(final_target, "/app/version"), b"A\n"); + assert_eq!( + docker + .inspect_volume(&resources.volume) + .await + .unwrap() + .created_at, + volume.created_at + ); + validate_committed_resource( + &fixture.config, + &resources.sandbox, + &docker.inspect_container(final_target, None).await.unwrap(), + LaunchValidation::Complete, + ) + .unwrap(); + // Explicitly remove only this stopped test workload and its admitted volume. + docker.remove_container(final_target, None).await.unwrap(); + docker + .remove_volume( + &resources.volume, + None::, + ) + .await + .unwrap(); +} diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index d65aeeed36..8bb5f9bbee 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -5,8 +5,10 @@ #![allow(clippy::result_large_err)] +mod image_update; mod isolation; pub mod otel_tracing; +mod workspace; use bollard::Docker; use bollard::errors::Error as BollardError; @@ -261,6 +263,10 @@ pub struct DockerComputeConfig { #[serde(default)] pub enable_bind_mounts: bool, + /// Give newly created default /sandbox workspaces dedicated persistent storage. + /// Existing writable-layer workspaces are not migrated by this setting. + pub persist_workspace: bool, + /// Corporate forward-proxy settings supplied to the supervisor. #[serde(flatten)] pub upstream_proxy: UpstreamProxyConfig, @@ -331,6 +337,7 @@ impl Default for DockerComputeConfig { ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(), sandbox_pids_limit: openshell_core::config::default_sandbox_pids_limit(), enable_bind_mounts: false, + persist_workspace: false, upstream_proxy: UpstreamProxyConfig::default(), proxy_ca_bundle: None, provider_spiffe_workload_api_socket: None, @@ -367,6 +374,7 @@ struct DockerDriverRuntimeConfig { gpu: DockerGpuRuntimeCapabilities, sandbox_pids_limit: Option, enable_bind_mounts: bool, + workspace_storage: workspace::Storage, upstream_proxy: UpstreamProxyConfig, proxy_ca_bundle: Option, provider_spiffe_workload_api_socket: Option, @@ -392,6 +400,7 @@ pub struct DockerComputeDriver { lifecycle_event_fences: DockerLifecycleEventFences, control_processes: Arc>>, runtime_failures: Arc>>, + image_update_gates: Arc, } struct DockerControlProcess { @@ -1031,6 +1040,11 @@ impl DockerComputeDriver { gpu, sandbox_pids_limit: docker_config.sandbox_pids_limit, enable_bind_mounts: docker_config.enable_bind_mounts, + workspace_storage: if docker_config.persist_workspace { + workspace::Storage::Persistent + } else { + workspace::Storage::WritableLayer + }, allow_driver_config: docker_config.allow_driver_config, resource_admission: docker_config.resource_admission.clone(), upstream_proxy: docker_config.upstream_proxy.clone(), @@ -1050,6 +1064,7 @@ impl DockerComputeDriver { lifecycle_event_fences: DockerLifecycleEventFences::default(), control_processes: Arc::new(Mutex::new(HashMap::new())), runtime_failures: Arc::new(Mutex::new(HashMap::new())), + image_update_gates: Arc::new(image_update::Gates::default()), }; Box::pin(driver.reconcile_runtime_resources_at_startup()) @@ -1100,7 +1115,14 @@ impl DockerComputeDriver { openshell_core::extension_protocol::ExtensionFamily::Compute, "openshell/docker", openshell_core::VERSION, - [], + if self.config.workspace_storage == workspace::Storage::Persistent { + vec![ + openshell_core::extension_protocol::COMPUTE_UPDATE_SANDBOX_CONFIGURATION + .to_string(), + ] + } else { + Vec::new() + }, )), } } @@ -1275,6 +1297,9 @@ impl DockerComputeDriver { &self, container: &bollard::models::ContainerInspectResponse, ) -> Result<(), Status> { + // Driver-owned state remains checked even when caller resource + // admission is explicitly disabled by an operator. + let persistent_workspace = workspace::admit(&self.docker, container, &self.config).await?; if self.config.allow_driver_config && !self.config.resource_admission.enabled { return Ok(()); } @@ -1334,6 +1359,12 @@ impl DockerComputeDriver { "sandbox-private volume ownership changed", )); } + } else if mount.destination.as_deref() + == Some(driver_mounts::DEFAULT_WORKSPACE_ROOT) + && persistent_workspace + { + // Independently checked above, including exact volume + // identity and the admitted workload UID/GID. } else if !expected.contains_key(name) && mount .destination @@ -1488,6 +1519,11 @@ impl DockerComputeDriver { if let Some(pending) = self.pending_snapshot(sandbox_id, sandbox_name).await? { return Ok(Some(pending)); } + if let Some(pending) = + image_update::pending_snapshot(&self.config, sandbox_id, sandbox_name)? + { + return Ok(Some(pending)); + } let container = self .find_managed_container_summary(sandbox_id, sandbox_name) .await?; @@ -1501,6 +1537,16 @@ impl DockerComputeDriver { Ok(None) } + async fn resolve_lifecycle_id(&self, id: &str, name: &str) -> Result, Status> { + if !id.is_empty() { + return Ok(Some(id.to_string())); + } + Ok(self + .get_sandbox_snapshot(id, name) + .await? + .map(|sandbox| sandbox.id)) + } + async fn current_snapshots(&self) -> Result, Status> { let containers = self.list_managed_container_summaries().await?; let mut container_sandboxes = Vec::with_capacity(containers.len()); @@ -1557,6 +1603,9 @@ impl DockerComputeDriver { // attached to both the sandbox and gateway. A running workload // container alone is not a usable sandbox. by_id.extend(self.pending_snapshot_map().await); + for pending in image_update::pending_snapshots(&self.config)? { + by_id.insert(pending.id.clone(), pending); + } let mut sandboxes = by_id.into_values().collect::>(); sandboxes.sort_by(|left, right| left.id.cmp(&right.id)); Ok(sandboxes) @@ -1685,14 +1734,14 @@ impl DockerComputeDriver { create_docker_channel_volume(&self.docker, sandbox, &self.config, true) .await .map_err(|status| { - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); DockerProvisioningFailure::new("BoundaryChannelCreateFailed", status.message()) })?; let token_file_created = match write_sandbox_token_file(sandbox, &self.config).await { Ok(created) => created, Err(status) => { cleanup_runtime_volumes().await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "SandboxTokenWriteFailed", status.message(), @@ -1701,7 +1750,7 @@ impl DockerComputeDriver { }; if !token_file_created { cleanup_runtime_volumes().await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "SandboxTokenWriteFailed", "Docker control mode requires a gateway sandbox token", @@ -1720,7 +1769,7 @@ impl DockerComputeDriver { Ok(devices) => devices, Err(status) => { cleanup_runtime_volumes().await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "ContainerCreateFailed", status.message(), @@ -1738,7 +1787,7 @@ impl DockerComputeDriver { Ok(body) => body, Err(status) => { cleanup_runtime_volumes().await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "ContainerCreateFailed", status.message(), @@ -1753,7 +1802,7 @@ impl DockerComputeDriver { Err(status) => { let _ = remove_docker_channel_volume_by_id(&self.docker, &sandbox.id, &self.config) .await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::from_admission_status(status)); } }; @@ -1766,6 +1815,25 @@ impl DockerComputeDriver { DockerProvisioningFailure::new("ResourceAdmissionDenied", error.to_string()) })?, ); + match workspace::prepare( + &self.docker, + sandbox, + &self.config, + &image, + &workload_identity, + ) + .await + { + Ok(_) => {} + Err(status) => { + cleanup_runtime_volumes().await; + cleanup_failed_docker_boundary_state(sandbox, &self.config); + return Err(DockerProvisioningFailure::from_status( + "WorkspaceStorageFailed", + status, + )); + } + } let create_result = async { openshell_otel::record_error_result( self.docker @@ -1792,7 +1860,7 @@ impl DockerComputeDriver { Ok(created) => created, Err(error) => { cleanup_runtime_volumes().await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::from_status( "ContainerCreateFailed", create_status_from_docker_error("create docker sandbox container", error), @@ -1810,7 +1878,7 @@ impl DockerComputeDriver { ) .await; cleanup_runtime_volumes().await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::from_status( "OuterFenceInspectFailed", internal_status("inspect Docker sandbox outer fence", error), @@ -1831,7 +1899,7 @@ impl DockerComputeDriver { ) .await; cleanup_runtime_volumes().await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::from_status( "OuterFenceRejected", status, @@ -1865,7 +1933,7 @@ impl DockerComputeDriver { ) .await; cleanup_runtime_volumes().await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( "BoundaryConfigWriteFailed", status.message(), @@ -1902,7 +1970,7 @@ impl DockerComputeDriver { "Failed to clean up Docker container after start failure" ); } - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); cleanup_runtime_volumes().await; return Err(DockerProvisioningFailure::from_status( "ContainerStartFailed", @@ -1939,7 +2007,7 @@ impl DockerComputeDriver { ) .await; cleanup_runtime_volumes().await; - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); return Err(DockerProvisioningFailure::new( supervisor_start_failure_reason(&status, "ControlSupervisorStartFailed"), status.message(), @@ -2098,7 +2166,7 @@ impl DockerComputeDriver { // leave a trusted supervisor eligible for launch. self.verify_network_trust()?; let sandboxes = self.list_managed_container_summaries().await?; - let sandbox_ids = sandboxes + let mut sandbox_ids = sandboxes .iter() .filter_map(|container| { container @@ -2108,6 +2176,9 @@ impl DockerComputeDriver { .cloned() }) .collect::>(); + // A valid accepted update can temporarily have no workload container. + // Its runtime channel/key resources are not orphans during that gap. + sandbox_ids.extend(image_update::pending_ids(&self.config)?); let filters = managed_resource_label_filters(&self.config.sandbox_namespace, []); let auxiliary = self @@ -2173,6 +2244,12 @@ impl DockerComputeDriver { .await .map_err(|error| internal_status("list Docker startup volumes", error))?; for volume in volumes.volumes.unwrap_or_default() { + // Workspace storage is retained independently of disposable runtime + // resources, including a future interrupted replacement gap. + if volume.labels.get(workspace::ROLE_LABEL).map(String::as_str) == Some(workspace::ROLE) + { + continue; + } let sandbox_id = volume.labels.get(LABEL_SANDBOX_ID); if sandbox_id.is_some_and(|id| sandbox_ids.contains(id)) { continue; @@ -2299,7 +2376,7 @@ impl DockerComputeDriver { { Ok(()) => { self.clear_runtime_failure(&record.sandbox.id).await; - remove_docker_channel_volume_by_id( + remove_docker_owned_storage_by_id( &self.docker, &record.sandbox.id, &self.config, @@ -2310,12 +2387,12 @@ impl DockerComputeDriver { } Err(err) if is_not_found_error(&err) => { self.clear_runtime_failure(&record.sandbox.id).await; - let _ = remove_docker_channel_volume_by_id( + remove_docker_owned_storage_by_id( &self.docker, &record.sandbox.id, &self.config, ) - .await; + .await?; cleanup_docker_boundary_state(&record.sandbox, &self.config); return Ok(true); } @@ -2325,14 +2402,17 @@ impl DockerComputeDriver { } } if !sandbox_id.is_empty() { + let removed_update = + image_update::pending_snapshot(&self.config, sandbox_id, sandbox_name)? + .is_some(); self.stop_control_process(sandbox_id).await; let removed = self .remove_auxiliary_containers_for_sandbox(sandbox_id) .await?; - remove_docker_channel_volume_by_id(&self.docker, sandbox_id, &self.config).await?; + remove_docker_owned_storage_by_id(&self.docker, sandbox_id, &self.config).await?; cleanup_docker_boundary_state_by_id(sandbox_id, &self.config); self.clear_runtime_failure(sandbox_id).await; - return Ok(removed); + return Ok(removed || removed_update); } return Ok(false); }; @@ -2358,19 +2438,15 @@ impl DockerComputeDriver { { Ok(()) => { self.clear_runtime_failure(resolved_sandbox_id).await; - remove_docker_channel_volume_by_id(&self.docker, resolved_sandbox_id, &self.config) + remove_docker_owned_storage_by_id(&self.docker, resolved_sandbox_id, &self.config) .await?; cleanup_docker_boundary_state_by_id(resolved_sandbox_id, &self.config); Ok(true) } Err(err) if is_not_found_error(&err) => { self.clear_runtime_failure(resolved_sandbox_id).await; - let _ = remove_docker_channel_volume_by_id( - &self.docker, - resolved_sandbox_id, - &self.config, - ) - .await; + remove_docker_owned_storage_by_id(&self.docker, resolved_sandbox_id, &self.config) + .await?; cleanup_docker_boundary_state_by_id(resolved_sandbox_id, &self.config); Ok(pending.is_some()) } @@ -2400,7 +2476,7 @@ impl DockerComputeDriver { self.clear_runtime_failure(&record.sandbox.id).await; remove_docker_channel_volume_by_id(&self.docker, &record.sandbox.id, &self.config) .await?; - cleanup_docker_boundary_state(&record.sandbox, &self.config); + cleanup_failed_docker_boundary_state(&record.sandbox, &self.config); self.publish_deleted(record.sandbox.id); return Ok(()); } @@ -2409,6 +2485,9 @@ impl DockerComputeDriver { self.remove_auxiliary_containers_for_sandbox(sandbox_id) .await?; } + if image_update::pending_snapshot(&self.config, sandbox_id, sandbox_name)?.is_some() { + return Ok(()); + } return Err(Status::not_found("sandbox not found")); }; let Some(target) = summary_container_target(&container) else { @@ -2478,18 +2557,28 @@ impl DockerComputeDriver { generation_id.to_string(), ) .map_err(|error| Status::invalid_argument(error.to_string()))?; + let resolved_id = self + .resolve_lifecycle_id(sandbox_id, sandbox_name) + .await? + .ok_or_else(|| Status::not_found("sandbox not found"))?; + let _lifecycle = self + .image_update_gates + .gate_for(&resolved_id) + .lock_owned() + .await; + image_update::require_settled(&self.config, &resolved_id)?; self.lifecycle_event_fences - .clear_stop(sandbox_id, sandbox_name); - self.lifecycle_event_fences.begin_start(sandbox_id); + .clear_stop(&resolved_id, sandbox_name); + self.lifecycle_event_fences.begin_start(&resolved_id); let result = Box::pin(self.start_sandbox_with_lifecycle_fence( - sandbox_id, + &resolved_id, sandbox_name, &generation, launch_authentication, durable_sandbox, )) .await; - self.lifecycle_event_fences.finish_start(sandbox_id); + self.lifecycle_event_fences.finish_start(&resolved_id); span_status.finish(result) } @@ -2519,6 +2608,18 @@ impl DockerComputeDriver { validate_docker_outer_fence(&inspected)?; self.admit_container_resources(&inspected).await?; let state = container.state.unwrap_or(ContainerSummaryStateEnum::EMPTY); + let validation = if container_state_needs_start(state) && !launch_authentication.is_empty() + { + image_update::LaunchValidation::BeforeStoppedRotation + } else { + image_update::LaunchValidation::Complete + }; + image_update::validate_committed_resource( + &self.config, + sandbox_id, + &inspected, + validation, + )?; let previous_finished_at = if state == ContainerSummaryStateEnum::EXITED { inspected .state @@ -2584,6 +2685,7 @@ impl DockerComputeDriver { ) .await?; } + image_update::validate_committed_boundary(&self.config, resolved_sandbox_id)?; let Some(runtime_descriptor) = read_docker_runtime_descriptor(resolved_sandbox_id, &self.config).await? else { @@ -2722,7 +2824,7 @@ impl DockerComputeDriver { sandbox: &DriverSandbox, failure: &DockerProvisioningFailure, ) { - cleanup_docker_boundary_state(sandbox, &self.config); + cleanup_failed_docker_boundary_state(sandbox, &self.config); let snapshot = pending_sandbox_snapshot( sandbox, &self.config.sandbox_namespace, @@ -2756,16 +2858,7 @@ impl DockerComputeDriver { sandbox_id: &str, sandbox_name: &str, ) -> Result<(), Status> { - if let Some(pending) = self.pending_snapshot(sandbox_id, sandbox_name).await? { - self.publish_sandbox_snapshot(pending); - return Ok(()); - } - if let Some(summary) = self - .find_managed_container_summary(sandbox_id, sandbox_name) - .await? - && let Some(mut sandbox) = sandbox_from_container_summary(&summary) - { - self.apply_runtime_failure(&mut sandbox).await; + if let Some(sandbox) = self.get_sandbox_snapshot(sandbox_id, sandbox_name).await? { self.publish_sandbox_snapshot(sandbox); } Ok(()) @@ -2976,7 +3069,7 @@ impl DockerComputeDriver { .await .map_err(|err| internal_status("find Docker sandbox container", err))?; - Ok(containers.into_iter().find(|summary| { + let mut matches = containers.into_iter().filter(|summary| { let Some(labels) = summary.labels.as_ref() else { return false; }; @@ -2992,7 +3085,14 @@ impl DockerComputeDriver { .get(LABEL_SANDBOX_NAME) .is_some_and(|value| value == sandbox_name); namespace_matches && id_matches && name_matches - })) + }); + let selected = matches.next(); + if matches.next().is_some() { + return Err(Status::failed_precondition( + "multiple managed Docker workloads match the native identity", + )); + } + Ok(selected) } async fn ensure_image_available( @@ -3145,9 +3245,6 @@ fn validate_docker_outer_fence( } Ok(()) } -#[tonic::async_trait] -impl openshell_core::compute::ComputeDriver for ComputeDriverService {} - #[tonic::async_trait] impl ComputeDriver for ComputeDriverService { type WatchSandboxesStream = WatchStream; @@ -3311,7 +3408,51 @@ impl ComputeDriver for ComputeDriverService { } #[tonic::async_trait] -impl openshell_core::compute::ComputeDriver for DockerComputeDriver {} +impl openshell_core::compute::ComputeDriver for DockerComputeDriver { + async fn update_sandbox_configuration( + &self, + request: Request, + ) -> Result< + Response, + Status, + > { + let span_status = openshell_otel::ErrorStatusGuard::current(); + let result = Box::pin(self.update_workload_image(&request.into_inner())).await; + span_status.finish(result.map(|()| { + Response::new(openshell_core::proto::compute::v1::UpdateSandboxConfigurationResponse {}) + })) + } +} + +#[tonic::async_trait] +impl openshell_core::compute::ComputeDriver for ComputeDriverService { + async fn update_sandbox_configuration( + &self, + request: Request, + ) -> Result< + Response, + Status, + > { + self.rpc_tracer + .trace( + openshell_otel::rpc::UPDATE_SANDBOX_CONFIGURATION, + openshell_core::compute::ComputeDriver::update_sandbox_configuration( + &self.driver, + request, + ), + ) + .await + } +} + +#[tonic::async_trait] +impl openshell_core::proto::compute::v1::compute_driver_configuration_updates_server::ComputeDriverConfigurationUpdates for ComputeDriverService { + async fn update_sandbox_configuration( + &self, request: Request, + ) -> Result, Status> { + openshell_core::compute::ComputeDriver::update_sandbox_configuration(self, request).await + } +} #[tonic::async_trait] impl ComputeDriver for DockerComputeDriver { @@ -3442,9 +3583,21 @@ impl ComputeDriver for DockerComputeDriver { request: Request, ) -> Result, Status> { let span_status = openshell_otel::ErrorStatusGuard::current(); - let request = request.into_inner(); + let mut request = request.into_inner(); require_sandbox_identifier(&request.sandbox_id, &request.name)?; + let Some(id) = self + .resolve_lifecycle_id(&request.sandbox_id, &request.name) + .await? + else { + return Err(Status::not_found("sandbox not found")); + }; + request.sandbox_id = id; + let _lifecycle = self + .image_update_gates + .gate_for(&request.sandbox_id) + .lock_owned() + .await; self.lifecycle_event_fences .request_stop(&request.sandbox_id, &request.name); if let Err(error) = self @@ -3503,9 +3656,21 @@ impl ComputeDriver for DockerComputeDriver { request: Request, ) -> Result, Status> { let span_status = openshell_otel::ErrorStatusGuard::current(); - let request = request.into_inner(); + let mut request = request.into_inner(); require_sandbox_identifier(&request.sandbox_id, &request.name)?; + let Some(id) = self + .resolve_lifecycle_id(&request.sandbox_id, &request.name) + .await? + else { + return Ok(Response::new(DeleteSandboxResponse { deleted: false })); + }; + request.sandbox_id = id; + let _lifecycle = self + .image_update_gates + .gate_for(&request.sandbox_id) + .lock_owned() + .await; let event_sandbox_id = request.sandbox_id.clone(); let deleted = self .delete_sandbox_inner(&request.sandbox_id, &request.name) @@ -4272,6 +4437,15 @@ async fn create_docker_runtime_volume( Ok(()) } +async fn remove_docker_owned_storage_by_id( + docker: &Docker, + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> Result<(), Status> { + workspace::remove(docker, sandbox_id, config).await?; + remove_docker_channel_volume_by_id(docker, sandbox_id, config).await +} + async fn remove_docker_channel_volume_by_id( docker: &Docker, sandbox_id: &str, @@ -4708,7 +4882,7 @@ async fn prepare_docker_boundary_files( let tls = generate_sandbox_tls_material(session_id) .map_err(|error| Status::internal(format!("generate Docker boundary TLS: {error}")))?; let verification_keys = gateway_verification_keys(&launch_authentication.verification_keys)?; - let provisioning = isolation::DockerBoundarySpec { + let mut provisioning = isolation::DockerBoundarySpec { boundary_id: sandbox.id.clone(), generation: launch_authentication .supervisor @@ -4738,6 +4912,19 @@ async fn prepare_docker_boundary_files( } .provision() .map_err(|error| Status::failed_precondition(error.to_string()))?; + if config.workspace_storage == workspace::Storage::Persistent { + let record = workspace::record_claim(&sandbox.id, config)?.ok_or_else(|| { + Status::failed_precondition( + "private Docker workspace record is missing during provisioning", + ) + })?; + for claims in [ + &mut provisioning.boundary_config.resource_claims, + &mut provisioning.runtime_descriptor.resource_claims, + ] { + claims.insert(workspace::RECORD_CLAIM.into(), record.clone()); + } + } let boundary_config = provisioning .boundary_config .encode() @@ -5933,9 +6120,23 @@ async fn stop_docker_control_process(mut process: DockerControlProcess) { let _ = process.task.await; } -/// Make a private, on-disk rollback copy of the driver state before a stopped -/// workload is replaced. Only regular files are copied: a compromised state -/// directory cannot smuggle symlinks into the gateway filesystem. +// Failed or interrupted creation can leave a retained workspace. Keep its +// independent ownership record until explicit deletion removes that volume. +fn cleanup_failed_docker_boundary_state( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, +) { + let Ok(path) = workspace::record_path(&sandbox.id, config) else { + return; + }; + match std::fs::symlink_metadata(path) { + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + cleanup_docker_boundary_state(sandbox, config); + } + _ => {} // Preserve unreadable/invalid records too; recovery fails closed. + } +} + fn cleanup_docker_boundary_state(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) { cleanup_docker_boundary_state_by_id(&sandbox.id, config); } @@ -6198,6 +6399,9 @@ fn build_container_create_body_for_image( ) .map_err(Status::failed_precondition)?; let mut user_mounts = docker_driver_mounts(driver_config)?; + if let Some(mount) = workspace::mount(sandbox, config, image)? { + user_mounts.push(mount); + } user_mounts.push(Mount { target: Some(BOUNDARY_MOUNT_PATH.to_string()), source: Some(docker_channel_volume_name(sandbox, config)), @@ -6231,6 +6435,8 @@ fn build_container_create_body_for_image( }] }); let mut labels = template.labels.clone(); + // Persistence is recorded in protected driver state, never inferred from + // or published over caller-owned template labels. labels.insert( "openshell.ai/private-image-volume-targets".into(), serde_json::to_string(&image.volumes) diff --git a/crates/openshell-driver-docker/src/main.rs b/crates/openshell-driver-docker/src/main.rs index 817b840d21..e15815c327 100644 --- a/crates/openshell-driver-docker/src/main.rs +++ b/crates/openshell-driver-docker/src/main.rs @@ -7,6 +7,7 @@ use std::path::PathBuf; use clap::Parser; use miette::{IntoDiagnostic, Result}; use openshell_core::VERSION; +use openshell_core::proto::compute::v1::compute_driver_configuration_updates_server::ComputeDriverConfigurationUpdatesServer; use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; use openshell_core::sandbox_env::SupervisorOtlpArgs; use openshell_driver_docker::{ComputeDriverService, DockerComputeConfig, DockerComputeDriver}; @@ -95,9 +96,11 @@ async fn main() -> Result<()> { let _cleanup = openshell_core::external_driver_socket::SocketCleanup::new(args.bind_socket.clone()); info!(socket = %args.bind_socket.display(), "Starting Docker compute driver"); + let service = ComputeDriverService::new(driver); tonic::transport::Server::builder() .layer(openshell_otel::compute_driver_rpc_layer()) - .add_service(ComputeDriverServer::new(ComputeDriverService::new(driver))) + .add_service(ComputeDriverServer::new(service.clone())) + .add_service(ComputeDriverConfigurationUpdatesServer::new(service)) .serve_with_incoming_shutdown( openshell_core::external_driver_socket::SameUidUnixIncoming::new(listener), shutdown_signal(), diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index c022adc923..40972a0c79 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -49,7 +49,7 @@ fn startup_error_log_tails_fit_grpc_header_budget() { assert!(format_log_tail("").is_empty()); } -fn test_launch_authentication() -> Vec { +pub fn test_launch_authentication() -> Vec { serde_json::to_vec(&SandboxLaunchAuthentication { supervisor: SupervisorAuthBundle { session_id: openshell_core::SandboxSessionId::new(), @@ -207,7 +207,7 @@ async fn restart_rotates_bootstrap_descriptor_and_provisioning_evidence_together fs::remove_dir_all(directory).unwrap(); } -fn test_sandbox() -> DriverSandbox { +pub fn test_sandbox() -> DriverSandbox { // Mirrors the gateway-supplied request: the public `Sandbox` API no // longer carries `namespace`, so the gateway elides the field and the // driver must source it from its own runtime config. @@ -329,7 +329,7 @@ fn gpu_resources(count: Option) -> ResourceRequirements { } } -fn runtime_config() -> DockerDriverRuntimeConfig { +pub fn runtime_config() -> DockerDriverRuntimeConfig { DockerDriverRuntimeConfig { // Existing lifecycle fixtures exercise the explicitly opted-out contract. allow_driver_config: true, @@ -361,6 +361,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig { }, sandbox_pids_limit: openshell_core::config::default_sandbox_pids_limit(), enable_bind_mounts: false, + workspace_storage: workspace::Storage::WritableLayer, upstream_proxy: UpstreamProxyConfig::default(), proxy_ca_bundle: None, provider_spiffe_workload_api_socket: None, @@ -531,7 +532,7 @@ fn sandbox_driver_config_cannot_override_proxy_ca_bundle() { assert!(error.message().contains("proxy_ca_bundle"), "{error}"); } -fn test_workload_identity() -> ResolvedWorkloadIdentity { +pub fn test_workload_identity() -> ResolvedWorkloadIdentity { ResolvedWorkloadIdentity::new( 1234, 1235, @@ -565,7 +566,7 @@ fn inspected_volume(driver: &str, options: HashMap) -> bollard:: } } -fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDriver { +pub fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDriver { let wsl_all_gpu_fallback_enabled = config.gpu.wsl_all_gpu_fallback_enabled; DockerComputeDriver { docker: Arc::new( @@ -582,6 +583,7 @@ fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDr lifecycle_event_fences: DockerLifecycleEventFences::default(), control_processes: Arc::new(Mutex::new(HashMap::new())), runtime_failures: Arc::new(Mutex::new(HashMap::new())), + image_update_gates: Arc::new(image_update::Gates::default()), } } @@ -2622,6 +2624,56 @@ fn managed_container_label_filters_include_gateway_namespace() { assert!(labels.contains(&format!("{LABEL_SANDBOX_ID}=sbx-123"))); } +#[test] +fn persistent_workspace_config_defaults_off_and_round_trips() { + let default: DockerComputeConfig = toml::from_str("").unwrap(); + assert!(!default.persist_workspace); + let enabled: DockerComputeConfig = toml::from_str("persist_workspace = true").unwrap(); + assert!(enabled.persist_workspace); + let saved = toml::to_string(&enabled).unwrap(); + assert!( + toml::from_str::(&saved) + .unwrap() + .persist_workspace + ); +} + +#[test] +fn persistent_workspace_create_mounts_only_the_owned_default_workspace() { + let sandbox = test_sandbox(); + let mut config = runtime_config(); + let old = build_container_create_body(&sandbox, &config).unwrap(); + assert!( + old.host_config + .unwrap() + .mounts + .unwrap() + .iter() + .all(|mount| mount.target.as_deref() != Some("/sandbox")) + ); + config.workspace_storage = workspace::Storage::Persistent; + let body = build_container_create_body(&sandbox, &config).unwrap(); + let mounts = body.host_config.unwrap().mounts.unwrap(); + let mounts = mounts + .iter() + .filter(|mount| mount.target.as_deref() == Some("/sandbox")) + .collect::>(); + assert_eq!(mounts.len(), 1); + assert_eq!(mounts[0].typ, Some(MountTypeEnum::VOLUME)); + assert_eq!(mounts[0].read_only, Some(false)); + assert_eq!( + mounts[0].volume_options.as_ref().unwrap().no_copy, + Some(true) + ); + assert!( + mounts[0] + .source + .as_ref() + .unwrap() + .starts_with("openshell-workspace-") + ); +} + #[test] fn build_container_create_body_replaces_inherited_cmd_with_sandbox_bootstrap() { let create_body = build_container_create_body(&test_sandbox(), &runtime_config()).unwrap(); @@ -3785,6 +3837,50 @@ fn build_container_create_body_limits_writable_runtime_storage_to_supervisor_ca( assert!(!tmpfs.contains_key("/run")); } +#[test] +fn legacy_workspace_template_labels_are_preserved() { + let mut sandbox = test_sandbox(); + let labels = &mut sandbox + .spec + .as_mut() + .unwrap() + .template + .as_mut() + .unwrap() + .labels; + labels.insert( + "openshell.ai/workspace-storage".into(), + "caller-layout".into(), + ); + labels.insert( + "openshell.ai/workspace-volume-identity".into(), + "caller-identity".into(), + ); + labels.insert( + "openshell.ai/docker-workspace-storage".into(), + "caller-value".into(), + ); + labels.insert( + "openshell.ai/docker-workspace-volume-identity".into(), + "not-json".into(), + ); + let expected = labels.clone(); + for storage in [ + workspace::Storage::WritableLayer, + workspace::Storage::Persistent, + ] { + let mut config = runtime_config(); + config.workspace_storage = storage; + let labels = build_container_create_body(&sandbox, &config) + .unwrap() + .labels + .unwrap(); + for (key, value) in &expected { + assert_eq!(labels.get(key), Some(value)); + } + } +} + #[test] fn build_container_create_body_uses_runtime_namespace_label() { // Regression test: the namespace label must come from the driver's diff --git a/crates/openshell-driver-docker/src/workspace.rs b/crates/openshell-driver-docker/src/workspace.rs new file mode 100644 index 0000000000..9c0cc1a85d --- /dev/null +++ b/crates/openshell-driver-docker/src/workspace.rs @@ -0,0 +1,806 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Opt-in workspace storage, separate from disposable runtime/channel volumes. + +use super::{ + DockerDriverRuntimeConfig, DockerImageMetadata, DriverSandbox, LABEL_ISOLATION_BACKEND, + LABEL_ISOLATION_BACKEND_OPEN_SHELL, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID, + LABEL_SANDBOX_NAMESPACE, LABEL_SANDBOX_WORKSPACE, docker_channel_volume_name_by_id, + docker_volume_identity, driver_mounts, is_not_found_error, remove_docker_volume, +}; +use bollard::Docker; +use bollard::models::{ + ContainerInspectResponse, Mount, MountTypeEnum, MountVolumeOptions, Volume, VolumeCreateRequest, +}; +use openshell_isolation_interface::contract::ResolvedWorkloadIdentity; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use tonic::Status; + +pub const ROLE_LABEL: &str = "openshell.ai/storage-role"; +pub const ROLE: &str = "workspace"; +pub const RECORD_FILE: &str = "workspace-storage.json"; +pub const RECORD_CLAIM: &str = "docker.workspace_storage"; +const MAX_RECORD_BYTES: u64 = 16 * 1024; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Record { + version: u32, + namespace: String, + sandbox_id: String, + uid: u32, + gid: u32, + volume_identity: serde_json::Value, +} + +pub fn record_path(id: &str, config: &DockerDriverRuntimeConfig) -> Result { + Ok(super::docker_boundary_state_dir_by_id(id, config)?.join(RECORD_FILE)) +} + +pub fn record_bytes( + id: &str, + config: &DockerDriverRuntimeConfig, + volume: &Volume, + identity: &ResolvedWorkloadIdentity, +) -> Result, Status> { + serde_json::to_vec(&Record { + version: 1, + namespace: config.sandbox_namespace.clone(), + sandbox_id: id.into(), + uid: identity.uid, + gid: identity.gid, + volume_identity: docker_volume_identity(volume), + }) + .map_err(|_| Status::internal("encode private Docker workspace record")) +} + +fn read_record(id: &str, config: &DockerDriverRuntimeConfig) -> Result, Status> { + let path = record_path(id, config)?; + let metadata = match std::fs::symlink_metadata(&path) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(_) => { + return Err(Status::failed_precondition( + "private Docker workspace record cannot be read", + )); + } + }; + if !metadata.is_file() || metadata.len() > MAX_RECORD_BYTES { + return Err(Status::failed_precondition( + "private Docker workspace record must be a bounded regular file", + )); + } + let record: Record = serde_json::from_slice(&std::fs::read(path).map_err(|_| { + Status::failed_precondition("private Docker workspace record cannot be read") + })?) + .map_err(|_| Status::failed_precondition("private Docker workspace record is invalid"))?; + if record.version != 1 + || record.namespace != config.sandbox_namespace + || record.sandbox_id != id + || record.uid == 0 + || record.gid == 0 + || record + .volume_identity + .get("name") + .and_then(serde_json::Value::as_str) + != Some(volume_name(id, config).as_str()) + { + return Err(Status::failed_precondition( + "private Docker workspace record belongs to another resource", + )); + } + Ok(Some(record)) +} + +pub fn record_claim( + id: &str, + config: &DockerDriverRuntimeConfig, +) -> Result, Status> { + read_record(id, config)? + .map(|record| { + serde_json::to_string(&record) + .map_err(|_| Status::internal("encode private Docker workspace claim")) + }) + .transpose() +} + +async fn load_record( + id: &str, + config: &DockerDriverRuntimeConfig, +) -> Result, Status> { + let record = read_record(id, config)?; + let descriptor = super::read_docker_runtime_descriptor(id, config).await?; + let claim = descriptor + .as_ref() + .and_then(|descriptor| descriptor.resource_claims.get(RECORD_CLAIM)); + match (&record, claim) { + (None, None) => return Ok(None), // Legacy container: labels are caller data. + (None, Some(_)) => { + return Err(Status::failed_precondition( + "private Docker workspace record is missing", + )); + } + (Some(record), Some(claim)) => { + if serde_json::to_string(record) + .map_err(|_| Status::internal("encode private Docker workspace claim"))? + != *claim + { + return Err(Status::failed_precondition( + "private Docker workspace record and descriptor disagree", + )); + } + } + (Some(_), None) if descriptor.is_some() => { + return Err(Status::failed_precondition( + "Docker descriptor lacks its private workspace claim", + )); + } + (Some(_), None) => {} // Fresh create: record precedes container/descriptor. + } + if let (Some(record), Some(descriptor)) = (&record, descriptor) + && (descriptor.boundary_id != id + || descriptor.workload_identity.uid != record.uid + || descriptor.workload_identity.gid != record.gid) + { + return Err(Status::failed_precondition( + "private Docker workspace runtime identity changed", + )); + } + Ok(record) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Storage { + WritableLayer, + Persistent, +} + +const UID_LABEL: &str = "openshell.ai/workspace-uid"; +const GID_LABEL: &str = "openshell.ai/workspace-gid"; + +fn volume_name(sandbox_id: &str, config: &DockerDriverRuntimeConfig) -> String { + docker_channel_volume_name_by_id(sandbox_id, config).replacen( + "openshell-channel-", + "openshell-workspace-", + 1, + ) +} + +pub fn mount( + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + image: &DockerImageMetadata, +) -> Result, Status> { + if config.workspace_storage != Storage::Persistent { + return Ok(None); + } + let root = driver_mounts::resolve_oci_workspace_root(&image.working_dir) + .map_err(Status::failed_precondition)?; + if root != driver_mounts::DEFAULT_WORKSPACE_ROOT { + return Err(Status::failed_precondition( + "persistent Docker workspaces currently require /sandbox", + )); + } + if image + .volumes + .iter() + .any(|target| Path::new(target).starts_with(&root) || Path::new(&root).starts_with(target)) + { + return Err(Status::failed_precondition( + "image-declared volumes overlap the persistent Docker workspace", + )); + } + Ok(Some(Mount { + target: Some(root), + source: Some(volume_name(&sandbox.id, config)), + typ: Some(MountTypeEnum::VOLUME), + read_only: Some(false), + volume_options: Some(MountVolumeOptions { + no_copy: Some(true), + ..Default::default() + }), + ..Default::default() + })) +} + +fn ownership_labels( + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> HashMap { + HashMap::from([ + (LABEL_MANAGED_BY.into(), LABEL_MANAGED_BY_VALUE.into()), + (LABEL_SANDBOX_ID.into(), sandbox_id.into()), + ( + LABEL_SANDBOX_NAMESPACE.into(), + config.sandbox_namespace.clone(), + ), + ( + LABEL_ISOLATION_BACKEND.into(), + LABEL_ISOLATION_BACKEND_OPEN_SHELL.into(), + ), + (ROLE_LABEL.into(), ROLE.into()), + ]) +} + +fn validate_volume( + volume: &Volume, + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> Result<(), Status> { + if volume.name != volume_name(sandbox_id, config) + || volume.driver != "local" + || !volume.options.is_empty() + || ownership_labels(sandbox_id, config) + .iter() + .any(|(key, value)| volume.labels.get(key) != Some(value)) + { + return Err(Status::failed_precondition( + "persistent Docker workspace ownership changed", + )); + } + Ok(()) +} + +pub async fn prepare( + docker: &Docker, + sandbox: &DriverSandbox, + config: &DockerDriverRuntimeConfig, + image: &DockerImageMetadata, + identity: &ResolvedWorkloadIdentity, +) -> Result, Status> { + if mount(sandbox, config, image)?.is_none() { + return Ok(None); + } + if read_record(&sandbox.id, config)?.is_some() { + return Err(Status::failed_precondition( + "persistent Docker workspace is already recorded; explicit recovery or removal is required", + )); + } + let name = volume_name(&sandbox.id, config); + // Initial creation must not adopt an existing data volume. Recovery/update + // will use the recorded volume identity, rather than this creation path. + match docker.inspect_volume(&name).await { + Ok(_) => { + return Err(Status::failed_precondition( + "persistent Docker workspace already exists; explicit recovery or removal is required", + )); + } + Err(error) if is_not_found_error(&error) => {} + Err(error) => { + return Err(Status::unavailable(format!( + "inspect persistent Docker workspace: {error}" + ))); + } + } + let mut labels = ownership_labels(&sandbox.id, config); + labels.insert(LABEL_SANDBOX_WORKSPACE.into(), sandbox.workspace.clone()); + labels.insert(UID_LABEL.into(), identity.uid.to_string()); + labels.insert(GID_LABEL.into(), identity.gid.to_string()); + let volume = docker + .create_volume(VolumeCreateRequest { + name: Some(name), + driver: Some("local".into()), + labels: Some(labels.clone()), + ..Default::default() + }) + .await + .map_err(|error| { + Status::unavailable(format!("create persistent Docker workspace: {error}")) + })?; + validate_volume(&volume, &sandbox.id, config)?; + if labels + .iter() + .any(|(key, value)| volume.labels.get(key) != Some(value)) + { + return Err(Status::failed_precondition( + "persistent Docker workspace initialization identity changed", + )); + } + super::prepare_docker_boundary_state_dir(sandbox, config)?; + let bytes = record_bytes(&sandbox.id, config, &volume, identity)?; + if bytes.len() as u64 > MAX_RECORD_BYTES { + return Err(Status::failed_precondition( + "private Docker workspace record exceeds its bound", + )); + } + super::image_update::durable_write(&record_path(&sandbox.id, config)?, &bytes)?; + Ok(Some(volume)) +} + +fn validate_attachment( + container: &ContainerInspectResponse, + volume: &Volume, + recorded: &serde_json::Value, + config: &DockerDriverRuntimeConfig, +) -> Result<(), Status> { + let labels = container + .config + .as_ref() + .and_then(|config| config.labels.as_ref()) + .ok_or_else(|| { + Status::failed_precondition("persistent Docker workspace lacks sandbox ownership") + })?; + let id = labels + .get(LABEL_SANDBOX_ID) + .filter(|id| !id.is_empty()) + .ok_or_else(|| { + Status::failed_precondition("persistent Docker workspace lacks sandbox ID") + })?; + validate_volume(volume, id, config)?; + let matching_mounts = container + .mounts + .as_deref() + .unwrap_or_default() + .iter() + .filter(|mount| mount.destination.as_deref() == Some(driver_mounts::DEFAULT_WORKSPACE_ROOT)) + .collect::>(); + let user = container + .config + .as_ref() + .and_then(|config| config.user.as_deref()); + let expected_user = match (volume.labels.get(UID_LABEL), volume.labels.get(GID_LABEL)) { + (Some(uid), Some(gid)) => format!("{uid}:{gid}"), + _ => { + return Err(Status::failed_precondition( + "persistent Docker workspace lacks workload identity", + )); + } + }; + if labels.get(LABEL_SANDBOX_NAMESPACE) != Some(&config.sandbox_namespace) + || labels.get(LABEL_SANDBOX_WORKSPACE) != volume.labels.get(LABEL_SANDBOX_WORKSPACE) + || user != Some(expected_user.as_str()) + || *recorded != docker_volume_identity(volume) + || matching_mounts.len() != 1 + || matching_mounts[0].typ != Some(bollard::models::MountPointTypeEnum::VOLUME) + || matching_mounts[0].name.as_deref() != Some(volume.name.as_str()) + || matching_mounts[0].rw != Some(true) + { + return Err(Status::failed_precondition( + "persistent Docker workspace attachment or identity changed", + )); + } + Ok(()) +} + +pub async fn admit( + docker: &Docker, + container: &ContainerInspectResponse, + config: &DockerDriverRuntimeConfig, +) -> Result { + let Some(id) = container + .config + .as_ref() + .and_then(|config| config.labels.as_ref()) + .and_then(|labels| labels.get(LABEL_SANDBOX_ID)) + .filter(|id| !id.is_empty()) + else { + return Ok(false); + }; + let Some(record) = load_record(id, config).await? else { + return Ok(false); + }; + if container + .config + .as_ref() + .and_then(|value| value.user.as_deref()) + != Some(format!("{}:{}", record.uid, record.gid).as_str()) + { + return Err(Status::failed_precondition( + "persistent Docker workspace workload identity changed", + )); + } + let name = volume_name(id, config); + let volume = docker.inspect_volume(&name).await.map_err(|error| { + Status::unavailable(format!("inspect retained Docker workspace: {error}")) + })?; + validate_attachment(container, &volume, &record.volume_identity, config)?; + Ok(true) +} + +pub async fn remove( + docker: &Docker, + sandbox_id: &str, + config: &DockerDriverRuntimeConfig, +) -> Result<(), Status> { + let Some(record) = load_record(sandbox_id, config).await? else { + return Ok(()); + }; + let name = volume_name(sandbox_id, config); + let volume = match docker.inspect_volume(&name).await { + Ok(volume) => volume, + Err(error) if is_not_found_error(&error) => return Ok(()), + Err(error) => { + return Err(Status::unavailable(format!( + "inspect Docker workspace for removal: {error}" + ))); + } + }; + validate_volume(&volume, sandbox_id, config)?; + if docker_volume_identity(&volume) != record.volume_identity { + return Err(Status::failed_precondition( + "persistent Docker workspace identity changed before removal", + )); + } + // Docker refuses removal while another container still uses the volume. + remove_docker_volume(docker, &name).await +} + +#[cfg(test)] +pub mod tests { + use super::*; + use crate::tests::{ + runtime_config, test_driver_with_config, test_sandbox, test_workload_identity, + }; + use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; + + // Exercise the real Bollard request/response path, without a Docker engine. + pub async fn docker_api( + replies: Vec<(&'static str, String, u16, serde_json::Value)>, + ) -> (Docker, tokio::task::JoinHandle>) { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let endpoint = format!("http://{}", listener.local_addr().unwrap()); + let task = tokio::spawn(async move { + let mut requests = Vec::new(); + for (method, suffix, status, response) in replies { + let (mut stream, _) = listener.accept().await.unwrap(); + let mut bytes = Vec::new(); + let (header_end, body_size) = loop { + let mut buffer = [0_u8; 4096]; + let count = stream.read(&mut buffer).await.unwrap(); + assert!(count > 0, "incomplete Docker request"); + bytes.extend_from_slice(&buffer[..count]); + assert!(bytes.len() < 65536, "oversized Docker request"); + if let Some(end) = bytes.windows(4).position(|v| v == b"\r\n\r\n") { + let headers = std::str::from_utf8(&bytes[..end]).unwrap(); + let size = headers + .lines() + .find_map(|line| { + let (key, value) = line.split_once(':')?; + key.eq_ignore_ascii_case("content-length") + .then(|| value.trim().parse::().unwrap()) + }) + .unwrap_or(0); + break (end + 4, size); + } + }; + while bytes.len() < header_end + body_size { + let mut buffer = [0_u8; 4096]; + let count = stream.read(&mut buffer).await.unwrap(); + assert!(count > 0); + bytes.extend_from_slice(&buffer[..count]); + } + let request_line = std::str::from_utf8(&bytes[..header_end]) + .unwrap() + .lines() + .next() + .unwrap(); + let fields = request_line.split_whitespace().collect::>(); + assert_eq!(fields[0], method); + assert!( + fields[1].ends_with(&suffix), + "unexpected path: {}", + fields[1] + ); + requests.push(if body_size == 0 { + serde_json::Value::Null + } else { + serde_json::from_slice(&bytes[header_end..header_end + body_size]).unwrap() + }); + let body = response.to_string(); + let reply = format!( + "HTTP/1.1 {status} Response\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + stream.write_all(reply.as_bytes()).await.unwrap(); + } + requests + }); + ( + Docker::connect_with_http(&endpoint, 2, bollard::API_DEFAULT_VERSION).unwrap(), + task, + ) + } + + fn fixture() -> (ContainerInspectResponse, Volume, DockerDriverRuntimeConfig) { + let mut config = runtime_config(); + config.sandbox_namespace = + format!("workspace-test-{}", openshell_core::SandboxSessionId::new()); + fixture_for(test_sandbox(), config) + } + + fn save_fixture_record(volume: &Volume, config: &DockerDriverRuntimeConfig) -> PathBuf { + let sandbox = test_sandbox(); + let directory = super::super::prepare_docker_boundary_state_dir(&sandbox, config).unwrap(); + super::super::image_update::durable_write( + &directory.join(RECORD_FILE), + &record_bytes(&sandbox.id, config, volume, &test_workload_identity()).unwrap(), + ) + .unwrap(); + directory + } + + pub fn fixture_for( + sandbox: DriverSandbox, + config: DockerDriverRuntimeConfig, + ) -> (ContainerInspectResponse, Volume, DockerDriverRuntimeConfig) { + let identity = test_workload_identity(); + let mut labels = ownership_labels(&sandbox.id, &config); + labels.insert(LABEL_SANDBOX_WORKSPACE.into(), sandbox.workspace.clone()); + labels.insert(UID_LABEL.into(), identity.uid.to_string()); + labels.insert(GID_LABEL.into(), identity.gid.to_string()); + let volume: Volume = serde_json::from_value(serde_json::json!({ + "Name": volume_name(&sandbox.id, &config), "Driver": "local", "Options": {}, + "Labels": labels, "Mountpoint": "/synthetic/workspace", "Scope": "local", "CreatedAt": "2026-11-10T00:00:00Z" + })).unwrap(); + let labels = HashMap::from([ + (LABEL_SANDBOX_ID.to_string(), sandbox.id), + ( + LABEL_SANDBOX_NAMESPACE.to_string(), + config.sandbox_namespace.clone(), + ), + (LABEL_SANDBOX_WORKSPACE.to_string(), sandbox.workspace), + ]); + let container = serde_json::from_value(serde_json::json!({ + "Config": {"Labels": labels, "User": format!("{}:{}", identity.uid, identity.gid)}, + "Mounts": [{"Type": "volume", "Name": volume.name, "Destination": "/sandbox", "RW": true}] + })).unwrap(); + (container, volume, config) + } + + #[tokio::test] + async fn legacy_workspace_labels_are_not_a_driver_storage_record() { + // These keys were previously accepted as arbitrary template labels. + // Their values, even persistent-v1 or invalid JSON, must not turn an + // existing writable-layer container into a driver-owned volume claim. + for labels in [ + serde_json::json!({"openshell.ai/workspace-storage":"persistent-v1"}), + serde_json::json!({"openshell.ai/workspace-volume-identity":"not-json"}), + serde_json::json!({"openshell.ai/workspace-storage":"persistent-v1", "openshell.ai/workspace-volume-identity":"not-json"}), + serde_json::json!({"openshell.ai/docker-workspace-storage":"persistent-v1", "openshell.ai/docker-workspace-volume-identity":"not-json"}), + ] { + let mut labels = labels; + labels[LABEL_SANDBOX_ID] = "sbx-123".into(); + let container = + serde_json::from_value(serde_json::json!({"Config":{"Labels":labels}})).unwrap(); + let (_, _, config) = fixture(); + let (docker, server) = docker_api(vec![]).await; + assert!(!admit(&docker, &container, &config).await.unwrap()); + server.await.unwrap(); + } + } + + #[tokio::test] + async fn legacy_removal_never_probes_or_removes_an_unrecorded_workspace() { + let (_, _, config) = fixture(); + let (docker, server) = docker_api(vec![]).await; + remove(&docker, "sbx-123", &config).await.unwrap(); + assert!(server.await.unwrap().is_empty()); + } + + #[test] + fn failed_creation_keeps_storage_authority_until_explicit_deletion() { + let (_, volume, config) = fixture(); + let sandbox = test_sandbox(); + let directory = save_fixture_record(&volume, &config); + super::super::cleanup_failed_docker_boundary_state(&sandbox, &config); + assert!(record_claim(&sandbox.id, &config).unwrap().is_some()); + super::super::cleanup_docker_boundary_state(&sandbox, &config); + assert!(!directory.exists()); + + let directory = super::super::prepare_docker_boundary_state_dir(&sandbox, &config).unwrap(); + super::super::cleanup_failed_docker_boundary_state(&sandbox, &config); + assert!(!directory.exists()); + } + + #[test] + fn retained_workspace_requires_original_volume_identity() { + let (container, mut volume, config) = fixture(); + let recorded = docker_volume_identity(&volume); + validate_attachment(&container, &volume, &recorded, &config).unwrap(); + volume.created_at = Some("2026-11-11T00:00:00Z".into()); + assert!(validate_attachment(&container, &volume, &recorded, &config).is_err()); + } + + #[test] + fn retained_workspace_rejects_foreign_backing_and_workload_user() { + let (mut container, mut volume, config) = fixture(); + let recorded = docker_volume_identity(&volume); + container.config.as_mut().unwrap().user = Some("2000:2000".into()); + assert!(validate_attachment(&container, &volume, &recorded, &config).is_err()); + volume + .options + .insert("device".into(), "/private/foreign".into()); + assert!(validate_volume(&volume, "sbx-123", &config).is_err()); + } + + #[test] + fn retained_workspace_rejects_unrecorded_mount_and_foreign_namespace() { + let (mut container, mut volume, config) = fixture(); + let recorded = docker_volume_identity(&volume); + volume + .labels + .insert(LABEL_SANDBOX_NAMESPACE.into(), "foreign".into()); + assert!(validate_volume(&volume, "sbx-123", &config).is_err()); + container.mounts = None; + assert!(validate_attachment(&container, &volume, &recorded, &config).is_err()); + } + + #[tokio::test] + async fn legacy_custom_workspace_mount_keeps_existing_admission_path() { + let container = serde_json::from_value(serde_json::json!({ + "Config": {"Labels": {}, "WorkingDir": "/project"}, + "Mounts": [{"Type":"volume", "Name":"external-data", "Destination":"/sandbox", "RW":true}] + })).unwrap(); + let (docker, server) = docker_api(vec![]).await; + assert!(!admit(&docker, &container, &runtime_config()).await.unwrap()); + server.await.unwrap(); + } + + #[test] + fn persistent_layout_is_opt_in_and_rejects_custom_workdir_and_nested_image_volume() { + let sandbox = test_sandbox(); + let mut config = runtime_config(); + let mut image = DockerImageMetadata { + id: "image".into(), + user: "1000:1000".into(), + working_dir: "/sandbox".into(), + volumes: vec![], + }; + assert!(mount(&sandbox, &config, &image).unwrap().is_none()); + config.workspace_storage = Storage::Persistent; + let persistent = mount(&sandbox, &config, &image).unwrap().unwrap(); + assert_eq!(persistent.volume_options.unwrap().no_copy, Some(true)); + image.working_dir = "/custom".into(); + assert!(mount(&sandbox, &config, &image).is_err()); + image.working_dir = "/sandbox".into(); + image.volumes.push("/sandbox/cache".into()); + assert!(mount(&sandbox, &config, &image).is_err()); + } + + #[tokio::test] + async fn creation_refuses_existing_data_without_relabeling_it() { + let (_, volume, mut config) = fixture(); + config.workspace_storage = Storage::Persistent; + let path = format!("/volumes/{}", volume.name); + let (docker, server) = docker_api(vec![( + "GET", + path, + 200, + serde_json::to_value(&volume).unwrap(), + )]) + .await; + let image = DockerImageMetadata { + id: "image".into(), + user: "1234:1235".into(), + working_dir: "/sandbox".into(), + volumes: vec![], + }; + let error = prepare( + &docker, + &test_sandbox(), + &config, + &image, + &test_workload_identity(), + ) + .await + .unwrap_err(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + assert_eq!(server.await.unwrap().len(), 1); + } + + #[tokio::test] + async fn creation_records_only_fresh_owned_workspace_storage() { + let (_, volume, mut config) = fixture(); + config.workspace_storage = Storage::Persistent; + let path = format!("/volumes/{}", volume.name); + let (docker, server) = docker_api(vec![ + ("GET", path, 404, serde_json::json!({"message":"not found"})), + ( + "POST", + "/volumes/create".into(), + 201, + serde_json::to_value(&volume).unwrap(), + ), + ]) + .await; + let image = DockerImageMetadata { + id: "image".into(), + user: "1234:1235".into(), + working_dir: "/sandbox".into(), + volumes: vec![], + }; + let created = prepare( + &docker, + &test_sandbox(), + &config, + &image, + &test_workload_identity(), + ) + .await + .unwrap() + .unwrap(); + assert_eq!( + docker_volume_identity(&created), + docker_volume_identity(&volume) + ); + let requests = server.await.unwrap(); + assert_eq!(requests[1]["Name"], volume.name); + assert_eq!(requests[1]["Driver"], "local"); + assert_eq!(requests[1]["Labels"][ROLE_LABEL], ROLE); + assert!(record_claim("sbx-123", &config).unwrap().is_some()); + std::fs::remove_dir_all(record_path("sbx-123", &config).unwrap().parent().unwrap()) + .unwrap(); + } + + #[tokio::test] + async fn disabling_external_admission_does_not_adopt_recreated_workspace_volume() { + let (container, mut volume, config) = fixture(); + assert!(config.allow_driver_config && !config.resource_admission.enabled); + let directory = save_fixture_record(&volume, &config); + volume.created_at = Some("2026-11-11T00:00:00Z".into()); + let (docker, server) = docker_api(vec![( + "GET", + format!("/volumes/{}", volume.name), + 200, + serde_json::to_value(&volume).unwrap(), + )]) + .await; + let mut driver = test_driver_with_config(config); + driver.docker = std::sync::Arc::new(docker); + let error = driver + .admit_container_resources(&container) + .await + .unwrap_err(); + assert_eq!(error.code(), tonic::Code::FailedPrecondition); + server.await.unwrap(); + std::fs::remove_dir_all(directory).unwrap(); + } + + #[tokio::test] + async fn removal_checks_ownership_and_never_forces_an_in_use_volume() { + let (_, mut volume, config) = fixture(); + let original = volume.clone(); + let directory = save_fixture_record(&volume, &config); + volume + .labels + .insert(LABEL_SANDBOX_NAMESPACE.into(), "foreign".into()); + let path = format!("/volumes/{}", volume.name); + let (docker, server) = docker_api(vec![( + "GET", + path.clone(), + 200, + serde_json::to_value(&volume).unwrap(), + )]) + .await; + assert_eq!( + remove(&docker, "sbx-123", &config) + .await + .unwrap_err() + .code(), + tonic::Code::FailedPrecondition + ); + server.await.unwrap(); + let volume = original; + let (docker, server) = docker_api(vec![ + ( + "GET", + path.clone(), + 200, + serde_json::to_value(&volume).unwrap(), + ), + ( + "DELETE", + path, + 409, + serde_json::json!({"message":"volume in use"}), + ), + ]) + .await; + assert!(remove(&docker, "sbx-123", &config).await.is_err()); + server.await.unwrap(); + std::fs::remove_dir_all(directory).unwrap(); + } +} diff --git a/crates/openshell-driver-docker/tests/fixtures/image-update/Dockerfile b/crates/openshell-driver-docker/tests/fixtures/image-update/Dockerfile new file mode 100644 index 0000000000..7ac1dce075 --- /dev/null +++ b/crates/openshell-driver-docker/tests/fixtures/image-update/Dockerfile @@ -0,0 +1,12 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +FROM alpine@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8 +ARG VERSION +ARG WORKLOAD_USER=1234:1235 +RUN mkdir -p /app \ + && printf '%s\n' "$VERSION" > /app/version \ + && printf 'other:x:2000:2000::/sandbox:/bin/sh\n' >> /etc/passwd \ + && printf 'other:x:2000:\n' >> /etc/group +USER ${WORKLOAD_USER} +WORKDIR /sandbox diff --git a/crates/openshell-gateway/src/lib.rs b/crates/openshell-gateway/src/lib.rs index 6f05eeaef5..2e75ec61f6 100644 --- a/crates/openshell-gateway/src/lib.rs +++ b/crates/openshell-gateway/src/lib.rs @@ -180,7 +180,7 @@ impl openshell_server::ComputeDriverFactory for MxcFactory { ) ))] fn install_in_tree_compute_drivers(registry: &mut ComputeDriverRegistry) { - for registration in [ + let registrations = [ #[cfg(feature = "compute-driver-kubernetes")] ComputeDriverRegistration::new( "kubernetes", @@ -221,7 +221,8 @@ fn install_in_tree_compute_drivers(registry: &mut ComputeDriverRegistry) { ComputeDriverRegistration::new("vm", u16::MAX, None, VmFactory).map(|registration| { registration.with_telemetry_category(TelemetryComputeDriver::anonymous_category("vm")) }), - ] { + ]; + for registration in registrations { registry .install(registration.expect("first-party driver name is valid")) .expect("first-party driver names are unique"); @@ -399,6 +400,18 @@ impl openshell_server::ComputeDriverFactory for DockerFactory { &self, context: openshell_server::ComputeDriverBuildContext<'_>, ) -> openshell_core::Result { + self.build_with_configuration_updates(context) + .await + .map(|(instance, _)| instance) + } + + async fn build_with_configuration_updates( + &self, + context: openshell_server::ComputeDriverBuildContext<'_>, + ) -> openshell_core::Result<( + openshell_server::ComputeDriverInstance, + Option, + )> { let mut config = docker_config(context.config_context())?; config.supervisor_otlp = context .otlp_config() @@ -415,9 +428,18 @@ impl openshell_server::ComputeDriverFactory for DockerFactory { ) .await .map_err(|error| openshell_core::Error::execution(error.to_string()))?; - let driver = openshell_driver_docker::ComputeDriverService::new_in_process(driver); - Ok(openshell_server::ComputeDriverInstance::InProcess( - std::sync::Arc::new(driver), + let driver = std::sync::Arc::new( + openshell_driver_docker::ComputeDriverService::new_in_process(driver), + ); + let updates: Option = + if config.persist_workspace { + Some(driver.clone()) + } else { + None + }; + Ok(( + openshell_server::ComputeDriverInstance::InProcess(driver), + updates, )) } } diff --git a/docs/how-it-works/gateways/configuration.mdx b/docs/how-it-works/gateways/configuration.mdx index a014615dd0..487585a91a 100644 --- a/docs/how-it-works/gateways/configuration.mdx +++ b/docs/how-it-works/gateways/configuration.mdx @@ -1533,6 +1533,9 @@ default_image = "nvcr.io/nvidia/base/ubuntu:24.04" image_pull_policy = "if_not_present" # Value assigned to the openshell.sandbox_namespace label on sandbox containers. sandbox_label = "docker-dev" +# Opt in newly created /sandbox workspaces to driver-owned persistent storage. +# Does not migrate existing workspaces. Updates also need enable_sandbox_updates. +persist_workspace = false # host (default) or bridge networking for the supervisor. # Operator-owned; sandbox driver config cannot override this choice. supervisor_network_mode = "host" @@ -1574,6 +1577,16 @@ provider_spiffe_workload_api_socket = "/run/spire/agent.sock" Use `sandbox_label` for Docker configurations. +`persist_workspace` defaults to `false`. When enabled, newly created workspaces +using the default `/sandbox` directory receive a driver-owned local volume. +Stop/start retains its data; deleting the sandbox removes the owned volume. +Custom working directories and image-declared volumes overlapping `/sandbox` +are rejected. Existing writable-layer workspaces remain unchanged. The driver +advertises image-only updates for persistent workspaces; the gateway also +requires `enable_sandbox_updates` before accepting them. See +[stopped sandbox image updates](/how-it-works/sandboxes/overview#update-the-image-of-a-stopped-sandbox) +for the supported image layout and update sequence. + `supervisor_network_mode` accepts `host` (the default), `bridge`, or `auto`. `auto` also selects host; the driver does not infer a mode from the workload runtime. The workload always uses `network=none`. diff --git a/docs/how-it-works/sandboxes/overview.mdx b/docs/how-it-works/sandboxes/overview.mdx index 9f61089f13..b2cb2f13a5 100644 --- a/docs/how-it-works/sandboxes/overview.mdx +++ b/docs/how-it-works/sandboxes/overview.mdx @@ -1027,8 +1027,9 @@ For a provisioning timeout, wait for cleanup to finish, then run `sandbox start` Use `openshell.v1.SandboxUpdates/UpdateSandbox` to change a stopped sandbox's workload image, command, environment overrides, labels, and annotations together. This separate service uses the same gateway listener and authentication as `OpenShell`; it requires `sandbox:write` and workspace -user access. This operation initially supports the Kubernetes compute driver -for workload changes; metadata-only updates work with every compute driver. There is no +user access. Kubernetes supports workload changes; Docker supports image-only +changes for newly created persistent `/sandbox` workspaces. Metadata-only updates +work with every compute driver. There is no `sandbox update` CLI command or curated SDK update method. Image updates default to disabled. Ask the operator to complete the @@ -1036,8 +1037,17 @@ Image updates default to disabled. Ask the operator to complete the before enabling `enable_sandbox_updates` in the [gateway configuration](/how-it-works/gateways/configuration#sandbox-updates). -The update preserves the sandbox ID, Kubernetes custom resource, and persistent -volume claim. It does not delete and recreate the sandbox, reseed its existing +The update preserves the sandbox ID and persistent workspace. Kubernetes retains +its custom resource and persistent volume claim; Docker replaces the stopped +workload container while retaining the owned workspace volume. Docker requires +`persist_workspace = true`, locally cached source and target images, and no +image-declared volumes. The target must declare an explicit numeric `USER UID:GID` +matching the recorded non-root workload identity; symbolic or implicit identities +are rejected before replacement. Existing writable-layer Docker +workspaces and custom working directories cannot be updated. Docker rejects +command and environment changes before acceptance. + +An update does not reseed its existing workspace from the new image, or change the trusted supervisor and sandbox runtime images. Omitted fields retain their current values. A supplied command replaces the argv list without shell expansion; an empty list restores the default diff --git a/docs/how-it-works/sandboxes/runtimes.mdx b/docs/how-it-works/sandboxes/runtimes.mdx index 70f6891ce1..d16938da01 100644 --- a/docs/how-it-works/sandboxes/runtimes.mdx +++ b/docs/how-it-works/sandboxes/runtimes.mdx @@ -119,6 +119,13 @@ socket_path = "/var/run/docker.sock" Common options in `[openshell.drivers.docker]` are `socket_path`, `grpc_endpoint`, `supervisor_network_mode`, `host_gateway_ip`, `sandbox_runtime_image`, `supervisor_image`, `image_pull_policy`, and `sandbox_pids_limit`. When `socket_path` is unset, the driver uses the socket found by auto-detection. +Set `persist_workspace = true` for driver-owned persistent storage in newly +created default `/sandbox` workspaces. Stop/start retains data; sandbox deletion +removes the owned volume. It defaults to `false`, does not migrate existing +workspaces. Image-only updates additionally require the gateway's +`enable_sandbox_updates` setting. See +[Docker configuration](/how-it-works/gateways/configuration#docker). + `supervisor_network_mode` defaults to `host`. Set `bridge` to use Docker bridge networking; `auto` also selects host. The workload always retains `network=none`. Host mode on Docker Desktop requires host networking enabled. Bridge mode automatically uses `host.docker.internal` for a gateway on the Desktop host. On native Linux, the gateway and host services must listen on bridge-reachable addresses. Set `grpc_endpoint` for remote gateways, remapped published ports, or other custom layouts. Endpoint selection preserves TLS verification and does not change the gateway's listener. See [Docker configuration](/how-it-works/gateways/configuration#docker). diff --git a/e2e/with-docker-gateway.sh b/e2e/with-docker-gateway.sh index ddcdad0f28..dd253f2ec6 100755 --- a/e2e/with-docker-gateway.sh +++ b/e2e/with-docker-gateway.sh @@ -126,6 +126,17 @@ ADDITIONAL_CA_SERVER_PID="" ADDITIONAL_CA_SERVER_LOG="${WORKDIR}/additional-ca-server.log" ADDITIONAL_CA_DIR="${WORKDIR}/additional-ca" ADDITIONAL_CA_PORT="" +IMAGE_UPDATES_MODE="${OPENSHELL_E2E_DOCKER_IMAGE_UPDATES:-0}" +SUPERVISOR_NETWORK_MODE="${OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE:-}" +case "${SUPERVISOR_NETWORK_MODE}" in + ""|host|bridge|auto) ;; + *) echo "ERROR: invalid Docker supervisor network mode for e2e." >&2; exit 2 ;; +esac + +if [ "${IMAGE_UPDATES_MODE}" = "1" ] && [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ]; then + echo "ERROR: Docker image-update qualification requires a new gateway owned by this wrapper." >&2 + exit 2 +fi if [ "${OIDC_MODE}" = "1" ] && [ -z "${OIDC_ISSUER}" ]; then echo "ERROR: OPENSHELL_E2E_OIDC_ISSUER is required when OPENSHELL_E2E_OIDC_GATEWAY=1" >&2 @@ -618,6 +629,9 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" { printf '[openshell]\nversion = 2\n\n' printf '[openshell.gateway]\nlog_level = "info"\n' + if [ "${IMAGE_UPDATES_MODE}" = "1" ]; then + printf 'enable_sandbox_updates = true\n' + fi printf 'guest_tls_ca = %s\n' "$(toml_string "${PKI_DIR}/ca.crt")" e2e_write_gateway_jwt_config "${JWT_DIR}" "openshell-e2e-docker-${HOST_PORT}" if [ "${OIDC_MODE}" != "1" ]; then @@ -644,6 +658,12 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" fi fi printf '[openshell.drivers.docker]\n' + if [ -n "${SUPERVISOR_NETWORK_MODE}" ]; then + printf 'supervisor_network_mode = %s\n' "$(toml_string "${SUPERVISOR_NETWORK_MODE}")" + fi + if [ "${IMAGE_UPDATES_MODE}" = "1" ]; then + printf 'persist_workspace = true\n' + fi printf 'allow_driver_config = true\n' printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")" # Container jobs need their bridge address; host runs exercise discovery. diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 319e2e6db4..dc908d4067 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -281,6 +281,18 @@ See [gateway configuration](https://docs.nvidia.com/openshell/latest/how-it-work ### Step 4: Check Docker-Backed Gateways +For stopped workload image updates, check both Docker `persist_workspace` and +gateway `enable_sandbox_updates`. Only newly created persistent `/sandbox` +workspaces support image-only changes; cached images must retain non-root +UID/GID and declare no volumes. An interrupted accepted update can temporarily +have no workload container. Retry the same gateway operation before Start; do +not treat that gap as an orphan or delete its volumes. The private driver update +record can contain environment and bootstrap material: do not export it in logs +or diagnostics. Workspace ownership requires the protected driver record and +matching runtime claim; caller container labels cannot enable persistence or +authorize volume adoption/removal. Preserve that record after failed creation +until explicit deletion. See the published sandbox update and Docker configuration docs. + The sandbox container's log holds the sandbox runtime's warnings and the main process's stdout and stderr when it runs without a TTY. The supervisor container's log holds supervisor diagnostics and OCSF shorthand. With diff --git a/skills/openshell-cli/SKILL.md b/skills/openshell-cli/SKILL.md index 1265b8a610..1be01aa663 100644 --- a/skills/openshell-cli/SKILL.md +++ b/skills/openshell-cli/SKILL.md @@ -583,16 +583,22 @@ complete the published [activation checklist](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability#activate-sandbox-updates) before enabling `enable_sandbox_updates`. Keep it false during a rolling upgrade and never enable it merely to bypass a failed compatibility check. -After activation, image, command, and environment updates initially support only -stopped Kubernetes sandboxes; other drivers return `UNIMPLEMENTED` for those -fields. Metadata-only updates also support stopped sandboxes on other drivers. Stop and +After activation, stopped Kubernetes sandboxes support image, command, and +environment updates. Docker supports image-only updates for workspaces created +with the driver's opt-in `persist_workspace` setting: both images must be cached, +the target must declare a numeric `USER UID:GID` matching the recorded non-root +identity, and neither image may declare volumes. Existing Docker writable-layer +workspaces are not migrated; command and environment updates remain unsupported. +Other drivers return `UNIMPLEMENTED` for configuration fields. Metadata-only +updates also support stopped sandboxes on other drivers. Stop and wait for `Stopped`, read the sandbox's current resource version through `GetSandbox`, call raw `UpdateSandbox` with the named workspace, sandbox name, selected image, command, environment overrides, labels or annotations, and required nonzero `expected_resource_version`, then start after success. Rust SDK callers use `client.raw_sandbox_updates_fresh()`; generated clients use `SandboxUpdates`, not the unchanged `OpenShell` service. There is no -curated SDK update method. The sandbox identity, Kubernetes -custom resource, and persistent volume claim remain; the existing workspace is +curated SDK update method. The sandbox identity and persistent workspace remain; +Kubernetes retains its custom resource and PVC, while Docker replaces the stopped +workload container and retains its owned `/sandbox` volume. The existing workspace is not reseeded and trusted runtime images do not change. Omitted update fields are unchanged; supplied lists and maps replace their user-owned values, including explicitly empty values. Template environment defaults still apply. System metadata diff --git a/tests/suites/features/Cargo.lock b/tests/suites/features/Cargo.lock index 0d91b3a2a9..d3a698f1e9 100644 --- a/tests/suites/features/Cargo.lock +++ b/tests/suites/features/Cargo.lock @@ -723,6 +723,17 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "docker-updates" +version = "0.0.0" +dependencies = [ + "openshell-testkit", + "serde_json", + "tokio", + "tonic", + "uuid", +] + [[package]] name = "dotenvy" version = "0.15.7" diff --git a/tests/suites/features/Cargo.toml b/tests/suites/features/Cargo.toml index a592b6f028..96ac6f00e5 100644 --- a/tests/suites/features/Cargo.toml +++ b/tests/suites/features/Cargo.toml @@ -3,4 +3,4 @@ [workspace] resolver = "2" -members = ["cdi", "host-gateway", "keycloak-provider-refresh", "kubernetes-updates", "oci-image"] +members = ["cdi", "docker-updates", "host-gateway", "keycloak-provider-refresh", "kubernetes-updates", "oci-image"] diff --git a/tests/suites/features/docker-updates/Cargo.toml b/tests/suites/features/docker-updates/Cargo.toml new file mode 100644 index 0000000000..abe4d704f3 --- /dev/null +++ b/tests/suites/features/docker-updates/Cargo.toml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "docker-updates" +description = "Authenticated Docker persistent-workspace image-update qualification" +version = "0.0.0" +edition = "2024" +rust-version = "1.94" +license = "Apache-2.0" +publish = false + +[dependencies] +openshell-testkit = { path = "../../../testkit/testkit" } +serde_json = "1" +tokio = { version = "1.43", features = ["process", "time"] } +tonic = "0.14" +uuid = { version = "1.10", features = ["v4"] } diff --git a/tests/suites/features/docker-updates/README.md b/tests/suites/features/docker-updates/README.md new file mode 100644 index 0000000000..6389d2999a --- /dev/null +++ b/tests/suites/features/docker-updates/README.md @@ -0,0 +1,50 @@ +# Docker image-update qualification + +This suite uses the real gateway, Docker driver, sandbox runtime and supervisor +through the public API and CLI. It requires a fresh gateway owned by the E2E +wrapper; it does not modify an existing gateway or Docker Desktop settings. + +It verifies: + +- Mutual TLS authorization, including rejection without the client identity. +- Workload execution through the SDK and authenticated SSH on images A and B. +- A stopped image-only update retaining the sandbox ID, SSH host fingerprint, + original workspace volume and a synthetic session record. +- Gateway restart and Stop/Start with authenticated workload access restored. +- Repair through Start of an intentionally interrupted credential publication + in the stopped test sandbox's private runtime state. +- Public API deletion removing the native workload, all four sandbox-owned + volumes and private runtime state. + +Build matching gateway, CLI and runtime images from the source under test. The +wrapper accepts `OPENSHELL_GATEWAY_BIN`, `OPENSHELL_BIN`, +`OPENSHELL_DOCKER_SANDBOX_RUNTIME_IMAGE` and +`OPENSHELL_DOCKER_SUPERVISOR_IMAGE` for prebuilt inputs. Record their source +revisions and hashes with the results. For local runtime images use +`mise run build:docker:sandbox build:docker:supervisor`. + +Prepare the A and B workload fixtures using the Dockerfile and instructions in +the [driver README](../../../../crates/openshell-driver-docker/README.md#live-driver-integration-test), +then run from the repository root: + +```shell +OPENSHELL_E2E_DOCKER_IMAGE_UPDATES=1 \ +OPENSHELL_E2E_DOCKER_SANDBOX_IMAGE=openshell-image-update-test:A \ +OPENSHELL_E2E_DOCKER_SANDBOX_IMAGE_PULL_POLICY=never \ +OPENSHELL_DOCKER_UPDATE_IMAGE_A=openshell-image-update-test:A \ +OPENSHELL_DOCKER_UPDATE_IMAGE_B=openshell-image-update-test:B \ +bash e2e/with-docker-gateway.sh \ + cargo test --locked --manifest-path tests/suites/features/Cargo.toml \ + -p docker-updates -- --nocapture +``` + +On Docker Desktop without host networking, also set +`OPENSHELL_E2E_DOCKER_SUPERVISOR_NETWORK_MODE=bridge`. Its documented +`host.docker.internal` endpoint can reach the host listener. Native Linux +loopback listeners use the default host mode. Select `DOCKER_HOST` to match +the Docker CLI context when necessary. TLS verification remains enabled. + +The fault injection changes only the descriptor of the stopped, test-owned +sandbox, after a successful launch. It does not print or export credential +files. The wrapper cleans up its private gateway state after the run. Preserve +a sanitized result summary and input hashes, rather than uploading raw state. diff --git a/tests/suites/features/docker-updates/tests/update.rs b/tests/suites/features/docker-updates/tests/update.rs new file mode 100644 index 0000000000..ae018d0af4 --- /dev/null +++ b/tests/suites/features/docker-updates/tests/update.rs @@ -0,0 +1,506 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Full gateway/supervisor qualification. Run only through the opt-in owned +//! gateway wrapper; authenticated API/exec, not direct driver test fixtures. + +use std::fs::{self, OpenOptions}; +use std::path::PathBuf; +use std::process::{Command, Stdio}; +use std::time::Duration; + +use serde_json::Value; +use testkit::sdk::raw::{AuthedGrpcClient, AuthedSandboxUpdatesClient, proto}; +use testkit::{Context, ResultExt as _}; + +const POLICY: &str = "version: 1 +filesystem_policy: + include_workdir: true + read_only: [/usr, /bin, /sbin, /lib, /lib64, /etc, /app, /proc, /dev/urandom] + read_write: [/tmp, /dev/null] +landlock: + compatibility: best_effort +process: + run_as_user: '1234' + run_as_group: '1235' +network_policies: {} +"; + +fn docker(args: &[&str]) -> Value { + let output = Command::new("docker") + .args(args) + .output() + .expect("Docker test inspection"); + assert!( + output.status.success(), + "Docker inspection failed: {args:?}" + ); + serde_json::from_slice(&output.stdout).expect("Docker JSON") +} + +fn container(sandbox_id: &str) -> Value { + let namespace = std::env::var("OPENSHELL_E2E_SANDBOX_NAMESPACE").expect("owned namespace"); + let output = Command::new("docker") + .args([ + "ps", + "-aq", + "--no-trunc", + "--filter", + &format!("label=openshell.ai/sandbox-id={sandbox_id}"), + "--filter", + &format!("label=openshell.ai/sandbox-namespace={namespace}"), + "--filter", + "label=openshell.ai/isolation-role=sandbox", + ]) + .output() + .unwrap(); + assert!(output.status.success()); + let ids = String::from_utf8(output.stdout).unwrap(); + let ids: Vec<_> = ids.lines().filter(|line| !line.is_empty()).collect(); + assert_eq!(ids.len(), 1, "one owned native workload"); + docker(&["inspect", ids[0]])[0].clone() +} + +fn workspace_volume(container: &Value) -> String { + let mounts: Vec<_> = container["Mounts"] + .as_array() + .unwrap() + .iter() + .filter(|mount| mount["Destination"] == "/sandbox" && mount["Type"] == "volume") + .collect(); + assert_eq!(mounts.len(), 1); + mounts[0]["Name"].as_str().unwrap().into() +} + +fn owned_volumes(sandbox_id: &str) -> Vec { + let output = Command::new("docker") + .args([ + "volume", + "ls", + "-q", + "--filter", + &format!("label=openshell.ai/sandbox-id={sandbox_id}"), + ]) + .output() + .unwrap(); + assert!(output.status.success()); + String::from_utf8(output.stdout) + .unwrap() + .lines() + .map(str::to_owned) + .collect() +} + +fn descriptor_path(sandbox_id: &str) -> PathBuf { + let root = PathBuf::from(std::env::var_os("XDG_STATE_HOME").expect("owned wrapper state")); + let pid = PathBuf::from(std::env::var_os("OPENSHELL_E2E_GATEWAY_PID_FILE").unwrap()); + assert_eq!( + root.parent(), + pid.parent(), + "fault injection is limited to the wrapper's private state" + ); + let mut pending = vec![(root, 0)]; + let mut matches = Vec::new(); + while let Some((directory, depth)) = pending.pop() { + assert!(depth < 8); + for entry in fs::read_dir(directory).unwrap() { + let entry = entry.unwrap(); + let metadata = entry.file_type().unwrap(); + if metadata.is_dir() { + pending.push((entry.path(), depth + 1)); + } else if metadata.is_file() && entry.file_name() == "runtime-descriptor.json" { + let value: Value = + serde_json::from_slice(&fs::read(entry.path()).unwrap()).unwrap(); + if value["boundary_id"] == sandbox_id { + matches.push(entry.path()); + } + } + } + } + assert_eq!(matches.len(), 1); + matches.remove(0) +} + +async fn get(client: &mut AuthedGrpcClient, workspace: &str, name: &str) -> proto::Sandbox { + client + .get_sandbox(proto::GetSandboxRequest { + workspace_scope: Some(proto::workspace_selector(workspace)), + name: name.into(), + }) + .await + .unwrap() + .into_inner() + .sandbox + .unwrap() +} + +async fn update( + client: &mut AuthedGrpcClient, + updates: &mut AuthedSandboxUpdatesClient, + workspace: &str, + name: &str, + id: &str, + image: &str, +) -> proto::Sandbox { + // Stop's final watch update can race its response. Retry only a definite + // CAS conflict with a fresh version/request ID, never ambiguous acceptance. + for attempt in 0..20 { + let current = get(client, workspace, name).await; + let metadata = current.metadata.unwrap(); + assert_eq!(metadata.id, id); + assert_eq!( + current.status.unwrap().phase, + proto::SandboxPhase::Stopped as i32 + ); + let mut request = tonic::Request::new(proto::UpdateSandboxRequest { + workspace_scope: Some(proto::workspace_selector(workspace)), + name: name.into(), + image: Some(image.into()), + expected_resource_version: metadata.resource_version, + request_id: uuid::Uuid::new_v4().to_string(), + ..Default::default() + }); + request.set_timeout(Duration::from_secs(70)); + match updates.update_sandbox(request).await { + Ok(response) => return response.into_inner().sandbox.unwrap(), + Err(error) => { + let conflict = error.code() == tonic::Code::Aborted + && testkit::sdk::SdkError::from_status(error.clone()) + .error_details() + .is_some_and(|details| { + details + .error_info() + .is_some_and(|info| info.reason == "RESOURCE_VERSION_CONFLICT") + }); + assert!(conflict && attempt < 19, "image update failed: {error}"); + tokio::time::sleep(Duration::from_millis(100)).await; + } + } + } + unreachable!() +} + +async fn restart_owned_gateway() { + let path = |key| PathBuf::from(std::env::var_os(key).expect("owned wrapper restart metadata")); + let binary = path("OPENSHELL_E2E_GATEWAY_BIN"); + let args_path = path("OPENSHELL_E2E_GATEWAY_ARGS_FILE"); + let pid_path = path("OPENSHELL_E2E_GATEWAY_PID_FILE"); + let log_path = path("OPENSHELL_E2E_GATEWAY_LOG"); + assert_eq!(args_path.parent(), pid_path.parent()); + assert_eq!(args_path.parent(), log_path.parent()); + let pid = fs::read_to_string(&pid_path) + .unwrap() + .trim() + .parse::() + .unwrap(); + let process = Command::new("ps") + .args(["-p", &pid.to_string(), "-o", "command="]) + .output() + .unwrap(); + assert!( + process.status.success() + && String::from_utf8_lossy(&process.stdout).contains(binary.to_str().unwrap()) + ); + assert!( + Command::new("kill") + .arg(pid.to_string()) + .status() + .unwrap() + .success() + ); + let mut stopped = false; + for _ in 0..120 { + let process = Command::new("ps") + .args(["-p", &pid.to_string(), "-o", "stat="]) + .output() + .unwrap(); + if !process.status.success() + || String::from_utf8_lossy(&process.stdout) + .trim() + .starts_with('Z') + { + stopped = true; + break; + } + tokio::time::sleep(Duration::from_millis(250)).await; + } + assert!(stopped, "owned gateway must stop normally; no force-kill"); + let args = fs::read(args_path).unwrap(); + let args: Vec<_> = args + .split(|byte| *byte == 0) + .filter(|part| !part.is_empty()) + .map(|part| std::str::from_utf8(part).unwrap()) + .collect(); + let log = OpenOptions::new().append(true).open(log_path).unwrap(); + let mut child = Command::new(binary) + .args(args) + .stdout(Stdio::from(log.try_clone().unwrap())) + .stderr(Stdio::from(log)) + .spawn() + .unwrap(); + fs::write(pid_path, format!("{}\n", child.id())).unwrap(); + // The outer wrapper owns shutdown through the updated PID file. Reap an + // early exit while the test is active without stopping the gateway before + // testkit has cleaned up any tracked sandbox after a failed assertion. + std::thread::spawn(move || { + let _ = child.wait(); + }); + testkit::Gateway::from_env() + .unwrap() + .wait_ready(Duration::from_secs(120)) + .await + .unwrap(); +} + +/// Authenticated workload use survives a stopped image replacement and gateway +/// restart with stable workspace/SSH identity, then API deletion removes data. +#[testkit::test] +async fn authenticated_replacement_restart_and_removal(cx: &mut Context) -> testkit::Result { + assert_eq!( + std::env::var("OPENSHELL_E2E_DOCKER_IMAGE_UPDATES").as_deref(), + Ok("1"), + "use the owned opt-in wrapper" + ); + // Require restart metadata before creating any resources. + std::env::var("OPENSHELL_E2E_GATEWAY_PID_FILE").expect("owned gateway required"); + cx.expect_compute_driver("docker").await?; + let selected = testkit::Gateway::from_env()?; + assert!( + selected.endpoint.starts_with("https://"), + "qualify authenticated TLS, not plaintext" + ); + assert!( + selected.describe().contains("(mTLS "), + "qualify a mutually authenticated gateway" + ); + let original_image = + std::env::var("OPENSHELL_DOCKER_UPDATE_IMAGE_A").expect("prepared image A"); + let target_image = std::env::var("OPENSHELL_DOCKER_UPDATE_IMAGE_B").expect("prepared image B"); + let gateway = selected.connect().await?; + let mut client = gateway.raw_grpc(); + let mut updates = gateway.raw_sandbox_updates(); + let sandbox = cx + .cli_sandbox("image") + .image(&original_image) + .policy_yaml(POLICY) + .command(["/bin/sh", "-c", "exec sleep 86400"]) + .create_ready() + .await?; + let original = get(&mut client, &selected.workspace, sandbox.name()).await; + let id = &original.metadata.as_ref().unwrap().id; + let mut unauthenticated_config = testkit::sdk::ClientConfig::new(&selected.endpoint); + unauthenticated_config.ca_cert = Some( + fs::read(std::env::var_os("OPENSHELL_E2E_GATEWAY_CA_CERT").expect("fixture CA")).unwrap(), + ); + let unauthenticated_rejected = + match testkit::sdk::OpenShellClient::connect(unauthenticated_config).await { + Err(_) => true, + Ok(anonymous) => anonymous + .raw_grpc() + .get_sandbox(proto::GetSandboxRequest { + workspace_scope: Some(proto::workspace_selector(&selected.workspace)), + name: sandbox.name().into(), + }) + .await + .is_err(), + }; + assert!( + unauthenticated_rejected, + "the same gateway must reject a client without its mTLS identity" + ); + assert_eq!( + get(&mut client, &selected.workspace, sandbox.name()) + .await + .metadata + .unwrap() + .id, + *id + ); + let fingerprint = &original.host_key_fingerprint; + assert!(fingerprint.starts_with("SHA256:")); + let before = container(id); + let volume = workspace_volume(&before); + let private_record: Value = serde_json::from_slice( + &fs::read(descriptor_path(id).with_file_name("workspace-storage.json")).unwrap(), + ) + .unwrap(); + assert_eq!(private_record["sandbox_id"], *id); + assert_eq!(private_record["volume_identity"]["name"], volume); + assert_eq!(private_record["uid"], 1234); + assert_eq!(private_record["gid"], 1235); + let volume_before = docker(&["volume", "inspect", &volume]); + sandbox.exec(["/bin/sh", "-c", "test \"$(id -u):$(id -g)\" = 1234:1235; cat /app/version; printf preserved-session > /sandbox/session-record"]).await?.expect_stdout("A\n")?; + cx.cli([ + "sandbox", + "exec", + sandbox.name(), + "--", + "/bin/cat", + "/sandbox/session-record", + ]) + .await? + .success()? + .expect_stdout("preserved-session")?; + sandbox.stop().await?; + let updated = update( + &mut client, + &mut updates, + &selected.workspace, + sandbox.name(), + id, + &target_image, + ) + .await; + assert_eq!(updated.metadata.as_ref().unwrap().id, *id); + assert_eq!(updated.host_key_fingerprint, *fingerprint); + let replaced = container(id); + assert_ne!(replaced["Id"], before["Id"]); + assert_ne!(replaced["Image"], before["Image"]); + assert_eq!(workspace_volume(&replaced), volume); + assert_eq!(replaced["State"]["Running"], false); + sandbox.start().await?; + sandbox.wait_ready().await?; + let descriptor = descriptor_path(id); + let previous_descriptor = fs::read(&descriptor).unwrap(); + sandbox + .exec([ + "/bin/sh", + "-c", + "cat /app/version; cat /sandbox/session-record", + ]) + .await? + .expect_stdout("B\npreserved-session")?; + cx.cli([ + "sandbox", + "exec", + sandbox.name(), + "--", + "/bin/cat", + "/app/version", + ]) + .await? + .success()? + .expect_stdout("B\n")?; + restart_owned_gateway().await; + sandbox + .wait_ready() + .await + .context("same sandbox ready after gateway restart")?; + sandbox + .exec([ + "/bin/sh", + "-c", + "cat /app/version; cat /sandbox/session-record", + ]) + .await? + .expect_stdout("B\npreserved-session")?; + cx.cli([ + "sandbox", + "exec", + sandbox.name(), + "--", + "/bin/cat", + "/sandbox/session-record", + ]) + .await? + .success()? + .expect_stdout("preserved-session")?; + let restarted = selected.connect().await?; + let recovered = get( + &mut restarted.raw_grpc(), + &selected.workspace, + sandbox.name(), + ) + .await; + assert_eq!(recovered.metadata.as_ref().unwrap().id, *id); + assert_eq!(recovered.host_key_fingerprint, *fingerprint); + assert_eq!(container(id)["Id"], replaced["Id"]); + assert_eq!( + docker(&["volume", "inspect", &volume])[0]["CreatedAt"], + volume_before[0]["CreatedAt"] + ); + sandbox.stop().await?; + sandbox.start().await?; + sandbox.wait_ready().await?; + sandbox + .exec(["/bin/cat", "/sandbox/session-record"]) + .await? + .expect_stdout("preserved-session")?; + sandbox.stop().await?; + let current: Value = serde_json::from_slice(&fs::read(&descriptor).unwrap()).unwrap(); + let previous: Value = serde_json::from_slice(&previous_descriptor).unwrap(); + assert_ne!(current["generation"], previous["generation"]); + // Recreate the reviewed interruption: the bootstrap is from a newer + // launch, but only the old descriptor survived. No running workload is + // changed, and neither credential file is exported or printed. + fs::write(&descriptor, &previous_descriptor).unwrap(); + sandbox.start().await?; + sandbox.wait_ready().await?; + sandbox + .exec(["/bin/cat", "/sandbox/session-record"]) + .await? + .expect_stdout("preserved-session")?; + cx.cli([ + "sandbox", + "exec", + sandbox.name(), + "--", + "/bin/cat", + "/app/version", + ]) + .await? + .success()? + .expect_stdout("B\n")?; + let repaired: Value = serde_json::from_slice(&fs::read(&descriptor).unwrap()).unwrap(); + let bootstrap: Value = serde_json::from_slice( + &fs::read(descriptor.with_file_name("boundary-bootstrap.json")).unwrap(), + ) + .unwrap(); + assert_ne!(repaired["generation"], previous["generation"]); + assert_eq!(repaired["generation"], bootstrap["generation"]); + assert_eq!(repaired["session_id"], bootstrap["session_id"]); + assert_eq!( + repaired["provisioning_evidence"], + bootstrap["provisioning_evidence"] + ); + let retained_volumes = owned_volumes(id); + assert!(retained_volumes.contains(&volume) && retained_volumes.len() >= 4); + sandbox.delete().await?; + sandbox.expect_absent().await?; + let native_id = replaced["Id"].as_str().unwrap(); + assert!( + !Command::new("docker") + .args(["container", "inspect", native_id]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status() + .unwrap() + .success() + ); + assert!( + !Command::new("docker") + .args(["volume", "inspect", &volume]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status() + .unwrap() + .success() + ); + assert!( + owned_volumes(id).is_empty(), + "API deletion removed all sandbox-owned volumes" + ); + assert!( + !descriptor.exists(), + "API deletion removed private runtime state" + ); + assert!( + !descriptor.parent().unwrap().exists(), + "API deletion removed the entire private sandbox directory" + ); + println!( + "qualification passed: authenticated mTLS API and SSH exec on A and B; stable sandbox/SSH identity and workspace; gateway restart; Stop/Start; interrupted credential-publication repair; API deletion removed native workload, all {} owned volumes and private runtime state", + retained_volumes.len() + ); + Ok(()) +}