Skip to content

bug(kubernetes): sandbox event indexes retain stale entries after watch relists #5596

Open

Description

@cloudonly

Update: Complete kind driver-watch relist reproduction and full harness now confirm stale A-event routing after recovery, with live-B and normal-delete-C controls. The production driver source is unchanged.

User Story

I use a kind-based OpenShell Kubernetes demo and inspect sandbox creation and runtime events. I need events to remain associated with the correct sandbox resources after a watch reconnect or resource replacement, so that old Pod events do not mislead diagnosis of a current sandbox.

Problem Statement

The shared-namespace Kubernetes watcher keeps its sandbox-name and Pod-name event indexes for the lifetime of the watch task. It handles InitApply like an ordinary update, but ignores both Init and InitDone. Updates only add entries. Entries are removed only when an explicit Delete arrives.

A kube-runtime relist therefore does not remove entries for resources absent from the new snapshot. This violates the dependency's snapshot contract: after InitDone, objects omitted from the InitApply set should be treated as deleted. A missed delete followed by relist can leave a stale event-correlation entry indefinitely within that watch task.

The event mapper also checks only involved-object kind/name, not involved-object UID. If a retained or reused name belongs to a different Pod, an event can still be attributed through the old association. Snapshot cleanup and UID-aware event attribution are related safeguards; this report does not assume that every old Pod alias is invalid, because delayed historical events may be intentional.

Impact / Why This Matters

A recovered watch can keep obsolete event routes and accumulate stale index entries. If an event names an obsolete or reused object, it can be routed to a sandbox stream using an association that no longer matches the Kubernetes object.

The demonstrated effect is event-index and attribution correctness. This report does not claim a workload isolation bypass, routine cross-tenant disclosure, or permanent gateway sandbox-record loss. The gateway has a separate reconciliation sweep for missing backend resources and closes event buffers for cleaned-up IDs.

Acceptance Criteria

  • After a completed relist, the active event indexes agree with the authoritative snapshot, including resources absent from that snapshot.
  • Events cannot be attributed to a live sandbox merely because a different Kubernetes object reused an indexed name.
  • Explicit deletion still removes the corresponding index entries.
  • An interrupted relist does not publish a partial index as a complete authoritative snapshot.
  • Tests cover an omitted resource after relist, resource replacement, delayed old-UID events, and normal current-UID events.
  • The intended retention policy for historical Pod aliases is explicit and covered by tests.

Reproduction Steps

Use an OpenShell deployment with the Kubernetes driver in shared workspace mode and an expendable sandbox.

  1. Create sandbox A and allow its Sandbox and Pod names to enter the driver's event indexes.
  2. During an interruption of the driver's Kubernetes watch, remove A. Allow the prior watch resourceVersion to expire so recovery performs a fresh list rather than resuming the old watch history.
  3. Restore watch access. The new list completes without A. The current code processes Init, the surviving objects' InitApply events, and InitDone without replacing or pruning its event indexes.
  4. A Kubernetes event naming A's old CR or Pod still resolves through the stale entry. A name-reuse test should use a different involved-object UID and verify that it is not attributed using the old association.

An ordinary brief disconnect may resume the previous watch rather than relist. Restarting the entire gateway also clears the maps and does not reproduce the retained-map case. Use an isolated test deployment when controlling the watch interruption; do not disrupt unrelated sandbox sessions.

A complete isolated driver-watch relist sequence has now been exercised against a real kind API: withheld DELETE, one injected 410, a real post-fault LIST containing only B, and a replacement watch using that list's resourceVersion. See the complete reproduction, controls, and harness.

Deterministic Verification

A temporary test compiled the current update_indexes, remove_indexes, and map_kube_event_to_platform helpers with lightweight data/timestamp fixtures. It exercised the same additive updates used by the current watch arms.

Apply A -> Init -> InitApply B -> InitDone
Result: A's resource-name and Pod-name entries remain.

Old sandbox ID -> reused Pod name
Pod event: same name, explicitly different UID
Result: event maps to the old sandbox ID.

Explicit Delete A
Result: all indexed aliases for A are removed.

Four helper behavior checks passed. This verifies the association behavior, not the complete async watch or a normal cross-sandbox exposure path.

kind baseline (initial verification)

A disposable sandbox was created with OpenShell 0.1.8 in shared workspace mode. A clearly marked synthetic Kubernetes Event referenced that sandbox's current workload Pod UID. A direct WatchSandbox subscription with follow_events=true and event_tail=200 received exactly that test marker from source kubernetes.

The sandbox was subsequently deleted, and the test Event still existed with the original Pod UID. This establishes a working deployment-level event-observation path and a retained old-object Event for the next test. That was only the initial event-observation baseline. The subsequent complete driver-watch relist test now confirms the Sandbox-name index defect using Suspended CR fixtures. Its test Events reference real Sandbox CR UIDs; it does not claim an additional live Pod-runtime or Gateway/CLI exposure result.

No third-party packages were installed: the API observer used the existing Node.js runtime's built-in HTTP/2 and TLS modules. Published evidence omits hostnames, internal addresses, private paths, and client credentials.

Environment

  • Source checked: 2e5589b2180dd432320adda124c4f7bcede119b5.
  • The same no-op relist arms and index helpers are present in v0.1.8.
  • kube-runtime: 0.99.0.
  • kind baseline: OpenShell CLI/gateway 0.1.8; Agent Sandbox controller v1.0.3; kind v0.33.0; Kubernetes v1.37.0; Linux x86_64.
  • Workspace mode: shared.

Code References

This is separate from #5028: that issue concerns missing Event forwarding in managed/operator mode, while this report concerns the indexes in the existing shared-mode Event forwarding path.

Activity

  1. cloudonly commented on Nov 16, 2026

    @cloudonly
    Author

    Complete driver-watch relist reproduction on kind

    Confirmed against a real kind API with the unmodified production openshell-driver-kubernetes library at 2e5589b2180dd432320adda124c4f7bcede119b5. The probe links that library and calls its public watch_sandboxes() entry point. It does not copy or substitute the watch loop, relist handling, readiness conversion, index functions, or event mapper.

    Exact scope

    • Dedicated test namespace with three explicitly marked, Suspended Agent Sandbox CR fixtures: A, B, and C. No workload Pod, PVC, or supervisor is launched.
    • Real Kubernetes LIST responses, ordinary watch notifications, and core/v1 Event objects.
    • One transport fault: withhold the first Sandbox watch's notifications while A is deleted, then inject one watch ERROR frame with code=410, reason=Expired. The subsequent recovery uses the actual kube-runtime state machine and actual API responses.
    • One driver initialization in the recorded run, with the same PID throughout. The existing Gateway and its workloads are not restarted or disconnected.
    • The proof is at the driver's outgoing observation stream. It does not assert Gateway/CLI exposure, permanent Gateway-record retention, or cross-tenant disclosure. The live witness is the Sandbox-name index; this case does not launch Pods to exercise the separate Pod-alias index.

    Environment: kind v0.33.0; Kubernetes v1.37.0; Agent Sandbox controller v1.0.3; kube-runtime 0.99.0. The existing Gateway is 0.1.8, but the producer under test is the linked driver library at the pinned source commit above.

    Observed sequence

    1. Initialize the original driver in an empty dedicated namespace, then create fixtures A and B. The driver receives both snapshots and opens Sandbox watch ci: add GitHub Actions CI workflow with lint, test, and image build #1.
    2. Pause delivery on that watch. Delete A through the real API. The transport records the real DELETED A notification and withholds it; the driver emits no Deleted(A).
    3. Inject the single 410 watch error. The same process re-lists through the real API. The returned list contains only B.
    4. The driver emits B again and opens watch Prototype data access layer that can connect to outlook data #2 using exactly the new list's resourceVersion. In kube-runtime's list/watch state machine, opening that watch follows completion of the relist/InitDone path. The harness does not fabricate Init, InitApply, or InitDone events.
    5. Create C, wait for its driver snapshot, then delete C normally. This time both the transport and driver record the ordinary DELETE.
    6. Create three clearly marked synthetic Kubernetes Events referencing the recorded, real UIDs of A, B, and C. A and C are already absent from the API. These are test Events, not claimed to be genuine controller/kubelet incidents.
    7. After the C Event exists, increment B's test Event count to 2 and wait for that update in the driver output as a final live-stream barrier.
    Target State before test Event DELETE delivered to driver Observed driver PlatformEvent
    A Deleted; absent from the completed relist No Forwarded to A — stale route confirmed
    B Still present Not applicable Forwarded normally, including the final count-2 update
    C Deleted normally after recovery Yes Not forwarded

    The late A Event leaves the original driver as source kubernetes, reason ReproAfterRelistA, sandbox ID fixture-a, even though A was omitted from the completed real list. C supplies the comparison: the normal DELETE branch removes that route. All three Event objects were independently checked in the actual API, including their references to the real captured UIDs; only B remained among the Sandbox CRs.

    Executed assertions

    The verifier passed all of these checks:

    {
      "driver_source_unchanged": true,
      "driver_initializations": 1,
      "driver_pid_unchanged": true,
      "injected_expired_watch_errors": 1,
      "withheld_real_delete_A": true,
      "other_proxy_rejections": 0,
      "real_relist_contains_only": ["B"],
      "second_watch_uses_relist_resource_version": true,
      "observed_B_again": true,
      "delivered_delete_A": false,
      "stale_A_forwarded": true,
      "live_B_forwarded": true,
      "normally_deleted_C_forwarded": false,
      "post_C_live_B_count": 2,
      "all_test_events_exist_in_real_api": true,
      "all_event_references_match_recorded_real_uids": true,
      "workload_pods": 0,
      "pvcs": 0
    }

    This closes the earlier gap between helper-only verification and a complete asynchronous driver-watch recovery test. It also keeps the attribution claim narrow: a stale driver route is demonstrated, while downstream cleanup/tombstone behavior remains a separate boundary.

    The isolated namespace and test Events were removed after the assertions. The observer, fault proxy, API proxy, and experiment tunnel were stopped. The original Gateway and existing sandbox remained Ready. Runtime PIDs, actual resource UIDs/resourceVersions, SSH aliases, host addresses, private paths, and client credentials are omitted from this report.

    Reproduction harness

    The expandable sections below contain the complete harness and step-by-step procedure. Save them under demo/issue-repros/5596/ in a checkout of the pinned source. Use an existing configured contributor toolchain, Node.js, kubectl, and a test kind cluster with the Agent Sandbox CRD. No additional packages need to be installed solely for this test.

    README.md
    # Issue 5596: complete driver-watch relist reproduction
    
    This case links the unmodified `openshell-driver-kubernetes` library and calls its public `watch_sandboxes()` entry point. It uses a real kind API for Sandbox CR lists, watch notifications, and Kubernetes Events. No watch/index helpers are copied into the probe.
    
    The transport withholds notifications on one Sandbox watch, then injects one explicitly synthetic watch `ERROR` frame with `code: 410`. The subsequent LIST response, its resourceVersion, the replacement watch, and the test Events all come from the real API. The original Gateway and its workloads are not reconfigured or restarted.
    
    The fixtures are Suspended Agent Sandbox CRs carrying OpenShell's managed-resource metadata. They intentionally have no workload Pods, PVCs, or supervisors. This tests the actual driver event producer and recovery state machine; it does not claim a Gateway/CLI exposure or a cross-tenant disclosure.
    
    ## Files
    
    - `probe.rs`: production driver constructor and watch stream observer.
    - `build-probe.mjs`: obtains the matching Cargo artifacts and links the probe.
    - `fault-proxy.mjs`: namespace-scoped GET-only transport and single-watch fault injection.
    - `namespace.yaml`, `fixtures.yaml`, `control-c.yaml`: fixed, isolated test resources.
    - `make-events.mjs`: marked synthetic Events referencing the real UIDs captured before deletion.
    - `verify-run.mjs`: asserts the trace, controls, and live API state; outputs a redacted summary.
    - `evidence.json`: redacted result of the completed run.
    
    ## Requirements
    
    Use the repository's configured Rust toolchain, an existing Node.js 20+ runtime, and kubectl access to a disposable kind environment with the Agent Sandbox v1beta1 CRD installed. No extra packages are installed by this case.
    
    Run from the OpenShell repository root. The examples assume the intended test kubeconfig/context is already selected. Check it explicitly. Use a fresh `os-relist-proof` namespace; do not adopt a namespace containing someone else's resources.
    
    ```shell
    kubectl config current-context
    kubectl get namespace os-relist-proof --ignore-not-found
    kubectl create --dry-run=server -o yaml -f demo/issue-repros/5596/namespace.yaml
    kubectl create -f demo/issue-repros/5596/namespace.yaml
    BUILD_DIR=$(mktemp -d)
    RUN_DIR=$(mktemp -d)
    node demo/issue-repros/5596/build-probe.mjs "$BUILD_DIR"
    ```
    
    In another terminal, start an existing-kubeconfig API proxy:
    
    ```shell
    kubectl proxy --address=127.0.0.1 --port=8001
    ```
    
    Start the observer in another terminal, retaining the chosen `BUILD_DIR` and `RUN_DIR` values:
    
    ```shell
    RELIST_UPSTREAM=http://127.0.0.1:8001 node demo/issue-repros/5596/fault-proxy.mjs "$BUILD_DIR/probe" "$RUN_DIR"
    ```
    
    Read `http://127.0.0.1:29182/__state` and wait for `driver_initialized` before creating fixtures. This keeps the constructor's existing-resource label backfill out of the case. The fixture gateway ID intentionally differs from the observer's gateway ID so its periodic lifecycle reconciliation does not own these fixtures. Its single-namespace event watcher still observes their OpenShell-managed metadata through the original production path.
    
    ```shell
    curl --noproxy '*' -fsS http://127.0.0.1:29182/__state
    kubectl create --dry-run=server -o yaml -f demo/issue-repros/5596/fixtures.yaml
    kubectl create -f demo/issue-repros/5596/fixtures.yaml
    export RELIST_UID_A=$(kubectl -n os-relist-proof get sandbox relist-a -o jsonpath='{.metadata.uid}')
    export RELIST_UID_B=$(kubectl -n os-relist-proof get sandbox relist-b -o jsonpath='{.metadata.uid}')
    curl --noproxy '*' -fsS -X POST http://127.0.0.1:29182/__probe/watch
    ```
    
    Wait for all three baseline conditions in `__state`: driver snapshots for `fixture-a` and `fixture-b`, plus `sandbox_watch_established` number 1. The timeline records the driver PID. Do not restart the observer during the fault sequence.
    
    ## Miss A's DELETE and complete a real relist
    
    ```shell
    curl --noproxy '*' -fsS -X POST http://127.0.0.1:29182/__inject/pause
    kubectl -n os-relist-proof get sandbox relist-a -o yaml
    kubectl -n os-relist-proof delete sandbox relist-a --wait=true --timeout=30s
    curl --noproxy '*' -fsS http://127.0.0.1:29182/__state
    ```
    
    Confirm `withheld_watch_event` contains the real API's `DELETED relist-a` and the driver has not emitted `deleted fixture-a`. Then inject the one expired-resourceVersion frame:
    
    ```shell
    curl --noproxy '*' -fsS -X POST http://127.0.0.1:29182/__inject/expire
    curl --noproxy '*' -fsS http://127.0.0.1:29182/__state
    ```
    
    Wait for a real post-injection Sandbox list containing only `relist-b`, another driver snapshot for B, and Sandbox watch number 2 using exactly that list's resourceVersion. In kube-runtime's list/watch state machine, opening that watch follows completion of the relist/`InitDone` path. No `Init`, `InitApply`, or `InitDone` events are fabricated by the harness.
    
    Normal watch renewals after this point may increase the watch number further. They do not restart the driver or invalidate the recorded second-watch barrier.
    
    ## Ordinary-delete control C
    
    ```shell
    kubectl create --dry-run=server -o yaml -f demo/issue-repros/5596/control-c.yaml
    kubectl create -f demo/issue-repros/5596/control-c.yaml
    export RELIST_UID_C=$(kubectl -n os-relist-proof get sandbox relist-c -o jsonpath='{.metadata.uid}')
    curl --noproxy '*' -fsS http://127.0.0.1:29182/__state
    ```
    
    Wait for the driver snapshot for C, then delete C without pausing notifications:
    
    ```shell
    kubectl -n os-relist-proof get sandbox relist-c -o yaml
    kubectl -n os-relist-proof delete sandbox relist-c --wait=true --timeout=30s
    curl --noproxy '*' -fsS http://127.0.0.1:29182/__state
    ```
    
    Wait for both `forwarded_watch_event: DELETED relist-c` and the driver's `deleted fixture-c` result.
    
    ## Late events and final live-stream barrier
    
    Keep the captured real UID environment variables. The generator fails if any is missing. Every generated Event is explicitly marked as a synthetic test fixture.
    
    ```shell
    node demo/issue-repros/5596/make-events.mjs "$RUN_DIR/events.json"
    kubectl create --dry-run=server -o yaml -f "$RUN_DIR/events.json"
    kubectl create -f "$RUN_DIR/events.json"
    kubectl -n os-relist-proof get events after-relist-a live-b-control deleted-c-control
    kubectl -n os-relist-proof patch event live-b-control --type=merge -p '{"count":2}' --dry-run=server -o yaml
    kubectl -n os-relist-proof patch event live-b-control --type=merge -p '{"count":2}'
    curl --noproxy '*' -fsS http://127.0.0.1:29182/__state
    ```
    
    Wait for the B platform event with metadata `count=2`. This is a positive live-stream barrier emitted after the C Event was created. Then verify the trace and the current real API state:
    
    ```shell
    RELIST_UPSTREAM=http://127.0.0.1:8001 node demo/issue-repros/5596/verify-run.mjs "$RUN_DIR/timeline.ndjson" "$RUN_DIR/events.json" "$RUN_DIR/evidence.json"
    ```
    
    Affected behavior:
    
    - A is absent from the completed relist, but `ReproAfterRelistA` still leaves the production driver as a platform event for `fixture-a`.
    - Live B's event is forwarded, including the final count-2 update.
    - C's event is not forwarded after the ordinary DELETE removed its index entry.
    
    The verifier also checks one driver initialization, unchanged PID, exactly one injected 410, no unrelated proxy rejections, the actual existence and UID references of all three test Events, only B remaining in the API, and zero workload Pods/PVCs.
    
    Raw timelines and generated Events contain runtime identifiers. Publish the verifier's redacted evidence summary rather than raw client configuration, raw Event manifests, or private environment details.
    
    ## Cleanup
    
    Stop the observer before deleting its namespace. Inspect the dedicated namespace and its fixed fixture resources before cleanup:
    
    ```shell
    curl --noproxy '*' -fsS -X POST http://127.0.0.1:29182/__stop
    kubectl get namespace os-relist-proof --show-labels
    kubectl -n os-relist-proof get sandbox,pods,pvc,events
    kubectl delete namespace os-relist-proof --wait=true --timeout=60s
    kubectl get namespace os-relist-proof --ignore-not-found
    ```
    
    Stop only the kubectl proxy and any tunnel created for this run. The original Gateway, its configuration, and existing sandbox workloads remain outside the test scope.
    probe.rs
    // SPDX-License-Identifier: Apache-2.0
    // Calls the production driver's public watch entry point; no copied watch helpers.
    use futures::StreamExt;
    use openshell_core::proto::compute::v1::watch_sandboxes_event::Payload;
    use openshell_driver_kubernetes::{KubernetesComputeConfig, KubernetesComputeDriver};
    use serde_json::json;
    use tokio::io::{AsyncBufReadExt, BufReader};
    
    #[tokio::main]
    async fn main() -> Result<(), Box<dyn std::error::Error>> {
        let namespace = std::env::var("RELIST_NAMESPACE")?;
        let (_shutdown_tx, shutdown_rx) = tokio::sync::watch::channel(false);
        let config = KubernetesComputeConfig {
            namespace,
            gateway_id: "relist-proof-observer".to_string(),
            ..Default::default()
        };
        let driver = KubernetesComputeDriver::new(config, shutdown_rx, None).await?;
        println!("{}", json!({"kind":"driver_initialized", "pid":std::process::id()}));
        let mut input = BufReader::new(tokio::io::stdin()).lines();
        if input.next_line().await?.as_deref() != Some("watch") {
            return Err("expected watch command after creating fixture CRs".into());
        }
        let mut stream = driver.watch_sandboxes().await?;
        println!("{}", json!({"kind":"watch_started", "pid":std::process::id()}));
        loop {
            tokio::select! {
                command = input.next_line() => {
                    match command? {
                        Some(line) if line == "quit" => break,
                        None => break,
                        _ => {}
                    }
                }
                event = stream.next() => {
                    let Some(event) = event else { break; };
                    match event?.payload {
                        Some(Payload::Sandbox(value)) => {
                            if let Some(sandbox) = value.sandbox {
                                println!("{}", json!({"kind":"sandbox", "id":sandbox.id, "name":sandbox.name}));
                            }
                        }
                        Some(Payload::Deleted(value)) => {
                            println!("{}", json!({"kind":"deleted", "id":value.sandbox_id}));
                        }
                        Some(Payload::PlatformEvent(value)) => {
                            if let Some(event) = value.event {
                                println!("{}", json!({"kind":"platform_event", "id":value.sandbox_id,
                                    "source":event.source, "reason":event.reason, "message":event.message,
                                    "metadata":event.metadata}));
                            }
                        }
                        None => {}
                    }
                }
            }
        }
        println!("{}", json!({"kind":"driver_stopped", "pid":std::process::id()}));
        Ok(())
    }
    build-probe.mjs
    import fs from 'node:fs';
    import path from 'node:path';
    import {fileURLToPath} from 'node:url';
    import {spawnSync} from 'node:child_process';
    const here=path.dirname(fileURLToPath(import.meta.url));
    const root=path.resolve(here,'../../..');
    const output=path.resolve(process.argv[2]??path.join(here,'build'));
    fs.mkdirSync(output,{recursive:true,mode:0o700});
    const build=spawnSync('cargo',['build','-p','openshell-driver-kubernetes','--lib','--locked','--message-format=json'],{cwd:root,encoding:'utf8',maxBuffer:16*1024*1024});
    if(build.status!==0){process.stderr.write(build.stderr??'');process.exit(build.status??1);}
    const libraries=new Map();
    for(const line of build.stdout.split('\n')){
     let item;try{item=JSON.parse(line);}catch{continue;}
     if(item.reason==='compiler-artifact'){
      const library=item.filenames?.find(name=>name.endsWith('.rlib'));
      if(library)libraries.set(item.target.name,library);
     }
    }
    const args=['--edition=2024',path.join(here,'probe.rs'),'-L',`dependency=${path.join(root,'target/debug/deps')}`,'-o',path.join(output,'probe')];
    for(const name of ['openshell_driver_kubernetes','openshell_core','tokio','futures','serde_json']){
     const library=libraries.get(name);if(!library)throw Error(`missing cargo artifact ${name}`);
     args.push('--extern',`${name}=${library}`);
    }
    const compile=spawnSync('rustc',args,{cwd:root,encoding:'utf8'});
    process.stdout.write(compile.stdout??'');process.stderr.write(compile.stderr??'');
    if(compile.status!==0)process.exit(compile.status??1);
    console.log(path.join(output,'probe'));
    fault-proxy.mjs
    import fs from 'node:fs';
    import path from 'node:path';
    import http from 'node:http';
    import {spawn} from 'node:child_process';
    
    const [binary,output]=process.argv.slice(2);
    if(!binary||!output)throw Error('usage: node fault-proxy.mjs PROBE_BINARY OUTPUT_DIRECTORY');
    const namespace=process.env.RELIST_NAMESPACE??'os-relist-proof';
    const upstream=new URL(process.env.RELIST_UPSTREAM??'http://127.0.0.1:29181');
    const port=Number(process.env.RELIST_PROXY_PORT??29182);
    if(!/^os-relist-proof(?:-[a-z0-9]+)?$/.test(namespace))throw Error('use a dedicated os-relist-proof namespace');
    fs.mkdirSync(output,{recursive:true,mode:0o700});
    const logFile=path.join(output,'timeline.ndjson');
    const records=[];let sequence=0;
    function record(data){const entry={seq:++sequence,time:new Date().toISOString(),...data};records.push(entry);fs.appendFileSync(logFile,JSON.stringify(entry)+'\n',{mode:0o600});console.log(JSON.stringify(entry));return entry;}
    let probe,probePid,watchNumber=0,paused=false,injected=false,activeSandbox;
    const pendingRequests=new Set();
    const agent=new http.Agent({keepAlive:true,maxSockets:32});
    const sandboxPath=`/apis/agents.x-k8s.io/v1beta1/namespaces/${namespace}/sandboxes`;
    const namespacePaths=[`/apis/agents.x-k8s.io/v1beta1/namespaces/${namespace}/`,`/apis/networking.k8s.io/v1/namespaces/${namespace}/`,`/api/v1/namespaces/${namespace}/`];
    function reply(response,status,data){response.writeHead(status,{'content-type':'application/json'});response.end(JSON.stringify(data));}
    const server=http.createServer(async(request,response)=>{
     const url=new URL(request.url,`http://127.0.0.1:${port}`);
     if(url.pathname==='/__state'){reply(response,200,{namespace,probePid,watchNumber,paused,injected,records});return;}
     if(url.pathname==='/__probe/watch'&&request.method==='POST'){
      if(!probe)throw Error('probe has not started');probe.stdin.write('watch\n');record({kind:'watch_command'});reply(response,200,{ok:true});return;
     }
     if(url.pathname==='/__inject/pause'&&request.method==='POST'){
      if(!activeSandbox||activeSandbox.closed){reply(response,409,{error:'no active Sandbox watch'});return;}
      paused=true;record({kind:'sandbox_watch_paused',watch:activeSandbox.number});reply(response,200,{ok:true});return;
     }
     if(url.pathname==='/__inject/expire'&&request.method==='POST'){
      if(!paused||injected||!activeSandbox||activeSandbox.closed){reply(response,409,{error:'pause one live watch before injecting once'});return;}
      const stream=activeSandbox;injected=true;paused=false;stream.closed=true;
      stream.response.write(JSON.stringify({type:'ERROR',object:{apiVersion:'v1',kind:'Status',status:'Failure',reason:'Expired',message:'OpenShell relist reproduction: injected expired resourceVersion',code:410}})+'\n');
      stream.response.end();stream.upstreamResponse.destroy();stream.upstreamRequest.destroy();
      record({kind:'injected_410',watch:stream.number});reply(response,200,{ok:true});return;
     }
     if(url.pathname==='/__stop'&&request.method==='POST'){
      record({kind:'stop_requested'});probe?.stdin.write('quit\n');reply(response,200,{ok:true});
      setTimeout(()=>{for(const r of pendingRequests)r.destroy();agent.destroy();server.closeAllConnections();server.close();if(probe&&!probe.killed)probe.kill('SIGTERM');},500);return;
     }
     const isSandbox=url.pathname===sandboxPath;
     const isWatch=url.searchParams.get('watch')==='true'||url.searchParams.get('watch')==='1';
     const allowed=url.pathname===`/api/v1/namespaces/${namespace}`||namespacePaths.some(prefix=>url.pathname.startsWith(prefix));
     if(request.method!=='GET'||!allowed){record({kind:'blocked_request',method:request.method,path:url.pathname});reply(response,403,{apiVersion:'v1',kind:'Status',reason:'Forbidden',message:'probe transport only permits GET in its isolated namespace',code:403});return;}
     const target=new URL(request.url,upstream);
     const upstreamRequest=http.request({hostname:target.hostname,port:target.port,path:target.pathname+target.search,method:'GET',agent,headers:{...request.headers,host:target.host}},upstreamResponse=>{
      const headers={...upstreamResponse.headers};delete headers['content-length'];delete headers['transfer-encoding'];delete headers.connection;
      response.writeHead(upstreamResponse.statusCode,headers);response.flushHeaders();
      let buffered='';
      let stream;
      if(isSandbox&&isWatch&&upstreamResponse.statusCode===200){
       stream={number:++watchNumber,response,upstreamResponse,upstreamRequest,closed:false};activeSandbox=stream;
       record({kind:'sandbox_watch_established',watch:stream.number,resourceVersion:url.searchParams.get('resourceVersion'),afterInjection:injected});
      }
      upstreamResponse.on('data',chunk=>{
       if(stream){
        if(stream.closed)return;
        buffered+=chunk.toString();let newline;
        while((newline=buffered.indexOf('\n'))!==-1){const line=buffered.slice(0,newline);buffered=buffered.slice(newline+1);if(!line.trim())continue;
         let event;try{event=JSON.parse(line);}catch{response.write(line+'\n');continue;}
         const name=event.object?.metadata?.name;
         if(paused){record({kind:'withheld_watch_event',watch:stream.number,type:event.type,name});continue;}
         if(['ADDED','MODIFIED','DELETED','ERROR'].includes(event.type))record({kind:'forwarded_watch_event',watch:stream.number,type:event.type,name});
         response.write(line+'\n');
        }
       }else{
        if(isSandbox&&!isWatch)buffered+=chunk.toString();
        response.write(chunk);
       }
      });
      upstreamResponse.on('end',()=>{
       if(stream){if(!stream.closed){if(buffered)response.write(buffered);response.end();stream.closed=true;}}
       else{
        if(isSandbox&&!isWatch){try{const list=JSON.parse(buffered);record({kind:'sandbox_list',status:upstreamResponse.statusCode,names:(list.items??[]).map(item=>item.metadata.name),resourceVersion:list.metadata?.resourceVersion,labelSelector:url.searchParams.get('labelSelector'),afterInjection:injected});}catch{}}
        response.end();
       }
      });
      upstreamResponse.on('error',error=>{if(!stream?.closed&&!response.destroyed){record({kind:'upstream_stream_error',message:error.message});response.destroy(error);}});
     });
     pendingRequests.add(upstreamRequest);
     upstreamRequest.on('close',()=>pendingRequests.delete(upstreamRequest));
     upstreamRequest.on('error',error=>{if(!response.destroyed&&!response.writableEnded){record({kind:'upstream_request_error',message:error.message});if(!response.headersSent)reply(response,502,{error:error.message});else response.destroy(error);}});
     response.on('close',()=>{if(!response.writableEnded)upstreamRequest.destroy();});
     upstreamRequest.end();
    });
    server.listen(port,'127.0.0.1',()=>{
     const kubeconfig=path.join(output,'probe-kubeconfig.yaml');
     fs.writeFileSync(kubeconfig,`apiVersion: v1\nkind: Config\nclusters:\n- name: relist-proof\n  cluster:\n    server: http://127.0.0.1:${port}\ncontexts:\n- name: relist-proof\n  context:\n    cluster: relist-proof\n    namespace: ${namespace}\n    user: relist-proof\ncurrent-context: relist-proof\nusers:\n- name: relist-proof\n  user: {}\n`,{mode:0o600});
     const env={...process.env,KUBECONFIG:kubeconfig,RELIST_NAMESPACE:namespace,NO_PROXY:'127.0.0.1,localhost'};
     delete env.KUBERNETES_SERVICE_HOST;delete env.KUBERNETES_SERVICE_PORT;
     probe=spawn(binary,[],{env,stdio:['pipe','pipe','pipe']});probePid=probe.pid;
     record({kind:'proxy_started',namespace,probePid});let text='';
     probe.stdout.on('data',chunk=>{text+=chunk;let newline;while((newline=text.indexOf('\n'))!==-1){const line=text.slice(0,newline);text=text.slice(newline+1);if(!line.trim())continue;try{record({origin:'driver',...JSON.parse(line)});}catch{record({kind:'driver_output',line});}}});
     probe.stderr.on('data',chunk=>record({kind:'driver_stderr',text:chunk.toString()}));
     probe.on('exit',(code,signal)=>record({kind:'probe_exit',code,signal}));
    });
    namespace.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: os-relist-proof
      labels:
        openshell.ai/reproduction: issue-5596
    fixtures.yaml
    # Suspended CR fixtures only: no workload, PVC, or supervisor is requested.
    apiVersion: agents.x-k8s.io/v1beta1
    kind: Sandbox
    metadata:
      name: relist-a
      namespace: os-relist-proof
      labels:
        openshell.ai/managed-by: openshell
        openshell.ai/sandbox-id: fixture-a
        openshell.ai/sandbox-name: relist-a
        openshell.ai/sandbox-workspace: default
        openshell.ai/gateway-id: relist-proof-fixtures
        openshell.ai/reproduction: issue-5596
    spec:
      operatingMode: Suspended
      podTemplate:
        spec:
          restartPolicy: Never
          containers:
            - name: agent
              image: busybox:1.36.1
              command: ["/bin/true"]
    ---
    apiVersion: agents.x-k8s.io/v1beta1
    kind: Sandbox
    metadata:
      name: relist-b
      namespace: os-relist-proof
      labels:
        openshell.ai/managed-by: openshell
        openshell.ai/sandbox-id: fixture-b
        openshell.ai/sandbox-name: relist-b
        openshell.ai/sandbox-workspace: default
        openshell.ai/gateway-id: relist-proof-fixtures
        openshell.ai/reproduction: issue-5596
    spec:
      operatingMode: Suspended
      podTemplate:
        spec:
          restartPolicy: Never
          containers:
            - name: agent
              image: busybox:1.36.1
              command: ["/bin/true"]
    control-c.yaml
    apiVersion: agents.x-k8s.io/v1beta1
    kind: Sandbox
    metadata:
      name: relist-c
      namespace: os-relist-proof
      labels:
        openshell.ai/managed-by: openshell
        openshell.ai/sandbox-id: fixture-c
        openshell.ai/sandbox-name: relist-c
        openshell.ai/sandbox-workspace: default
        openshell.ai/gateway-id: relist-proof-fixtures
        openshell.ai/reproduction: issue-5596
    spec:
      operatingMode: Suspended
      podTemplate:
        spec:
          restartPolicy: Never
          containers:
            - name: agent
              image: busybox:1.36.1
              command: ["/bin/true"]
    make-events.mjs
    import fs from 'node:fs';
    const output=process.argv[2];
    if(!output)throw Error('usage: RELIST_UID_A=... RELIST_UID_B=... RELIST_UID_C=... node make-events.mjs OUTPUT.json');
    const namespace=process.env.RELIST_NAMESPACE??'os-relist-proof';
    const cases=[['a','after-relist-a','ReproAfterRelistA'],['b','live-b-control','ReproLiveB'],['c','deleted-c-control','ReproDeletedC']];
    const timestamp=new Date().toISOString();
    const items=cases.map(([letter,name,reason])=>{
     const uid=process.env[`RELIST_UID_${letter.toUpperCase()}`];
     if(!uid||!/^[0-9a-f-]{36}$/.test(uid))throw Error(`set the recorded real UID for ${letter}`);
     return {apiVersion:'v1',kind:'Event',metadata:{name,namespace,labels:{'openshell.ai/reproduction':'issue-5596'}},
      involvedObject:{apiVersion:'agents.x-k8s.io/v1beta1',kind:'Sandbox',name:`relist-${letter}`,namespace,uid},
      reason,message:`Synthetic issue-5596 control event for Sandbox ${letter.toUpperCase()}`,
      source:{component:'openshell-relist-proof'},type:'Normal',firstTimestamp:timestamp,lastTimestamp:timestamp,count:1};
    });
    fs.writeFileSync(output,JSON.stringify({apiVersion:'v1',kind:'List',items},null,2)+'\n',{mode:0o600});
    console.log(JSON.stringify({eventNames:items.map(item=>item.metadata.name),namespace,synthetic:true}));
    verify-run.mjs
    import fs from 'node:fs';
    import path from 'node:path';
    import assert from 'node:assert/strict';
    import {spawnSync} from 'node:child_process';
    import {fileURLToPath} from 'node:url';
    
    const [timelineFile,eventFixtureFile,outputFile]=process.argv.slice(2);
    if(!outputFile)throw Error('usage: node verify-run.mjs TIMELINE.ndjson PRIVATE_EVENTS.json EVIDENCE.json');
    const records=fs.readFileSync(timelineFile,'utf8').trim().split('\n').map(line=>JSON.parse(line));
    const isDriver=(r,kind)=>r.origin==='driver'&&r.kind===kind;
    const initialized=records.filter(r=>r.kind==='driver_initialized');
    assert.equal(initialized.length,1,'driver must initialize only once');
    assert.equal(records.find(r=>r.kind==='watch_started')?.pid,initialized[0].pid,'driver PID must remain unchanged');
    const faults=records.filter(r=>r.kind==='injected_410');assert.equal(faults.length,1);
    const fault=faults[0];
    assert(records.some(r=>r.kind==='withheld_watch_event'&&r.type==='DELETED'&&r.name==='relist-a'&&r.seq<fault.seq));
    assert(!records.some(r=>isDriver(r,'deleted')&&r.id==='fixture-a'));
    for(const id of ['fixture-a','fixture-b'])assert(records.some(r=>isDriver(r,'sandbox')&&r.id===id&&r.seq<fault.seq));
    const relist=records.find(r=>r.kind==='sandbox_list'&&r.afterInjection&&r.labelSelector&&!r.labelSelector.includes('gateway-id')&&JSON.stringify(r.names)==='["relist-b"]');
    assert(relist,'a real post-fault list must contain only B');
    const resumed=records.find(r=>r.kind==='sandbox_watch_established'&&r.watch===2&&r.seq>relist.seq);
    assert(resumed,'second real watch must follow the relist');
    assert.equal(resumed.resourceVersion,relist.resourceVersion,'second watch must use the completed list resourceVersion');
    assert(records.some(r=>isDriver(r,'sandbox')&&r.id==='fixture-b'&&r.seq>fault.seq));
    const deletedC=records.find(r=>isDriver(r,'deleted')&&r.id==='fixture-c');assert(deletedC);
    assert(records.some(r=>r.kind==='forwarded_watch_event'&&r.type==='DELETED'&&r.name==='relist-c'&&r.seq<deletedC.seq));
    const platformEvents=records.filter(r=>isDriver(r,'platform_event'));
    const stale=platformEvents.find(r=>r.id==='fixture-a'&&r.reason==='ReproAfterRelistA'&&r.seq>resumed.seq);assert(stale,'stale A event must actually leave the original driver');
    assert(platformEvents.some(r=>r.id==='fixture-b'&&r.reason==='ReproLiveB'&&r.metadata.count==='1'));
    const barrier=platformEvents.find(r=>r.id==='fixture-b'&&r.reason==='ReproLiveB'&&r.metadata.count==='2'&&r.seq>deletedC.seq);assert(barrier,'live event barrier after the C case must arrive');
    assert(!platformEvents.some(r=>r.reason==='ReproDeletedC'),'normal DELETE must remove the C event route');
    assert.equal(records.filter(r=>r.kind==='blocked_request').length,0,'no unrelated proxy rejections may contaminate the run');
    const namespace=process.env.RELIST_NAMESPACE??'os-relist-proof';
    const upstream=process.env.RELIST_UPSTREAM??'http://127.0.0.1:29181';
    async function api(url){const response=await fetch(new URL(url,upstream),{signal:AbortSignal.timeout(10000)});assert.equal(response.status,200);return response.json();}
    const [sandboxes,events,pods,pvcs]=await Promise.all([
     api(`/apis/agents.x-k8s.io/v1beta1/namespaces/${namespace}/sandboxes`),
     api(`/api/v1/namespaces/${namespace}/events`),
     api(`/api/v1/namespaces/${namespace}/pods`),
     api(`/api/v1/namespaces/${namespace}/persistentvolumeclaims`)
    ]);
    assert.deepEqual(sandboxes.items.map(item=>item.metadata.name),['relist-b']);
    assert.equal(sandboxes.items[0].spec.operatingMode,'Suspended');
    assert.equal(pods.items.length,0);assert.equal(pvcs.items.length,0);
    const fixtures=JSON.parse(fs.readFileSync(eventFixtureFile,'utf8')).items;
    for(const expected of fixtures){const actual=events.items.find(item=>item.metadata.name===expected.metadata.name);assert(actual,`real event ${expected.metadata.name} must exist`);assert.equal(actual.involvedObject.uid,expected.involvedObject.uid);}
    assert.equal(events.items.find(item=>item.metadata.name==='live-b-control').count,2);
    const root=path.resolve(path.dirname(fileURLToPath(import.meta.url)),'../../..');
    const diff=spawnSync('git',['diff','--quiet','HEAD','--','crates/openshell-driver-kubernetes'],{cwd:root});assert.equal(diff.status,0,'production Kubernetes driver sources must remain unchanged');
    const head=spawnSync('git',['rev-parse','HEAD'],{cwd:root,encoding:'utf8'}).stdout.trim();
    const evidence={source_commit:head,driver_source_unchanged:true,driver_initializations:1,driver_pid_unchanged:true,
     fault:{injected_expired_watch_errors:1,withheld_real_delete_A:true,other_proxy_rejections:0},
     recovery:{real_list_contains_only:['B'],second_watch_uses_relist_resource_version:true,observed_B_again:true,delivered_delete_A:false},
     event_results:{stale_A_forwarded:true,live_B_forwarded:true,normally_deleted_C_forwarded:false,post_C_live_B_count:2},
     real_api_checks:{remaining_sandboxes:['B'],all_test_events_exist:true,all_event_references_match_recorded_real_uids:true,workload_pods:0,pvcs:0},
     scope:'Unmodified production driver watch entry point against a real kind API; one isolated watch fault was injected. No main gateway/CLI exposure claim.'};
    fs.writeFileSync(outputFile,JSON.stringify(evidence,null,2)+'\n',{mode:0o600});
    console.log(JSON.stringify(evidence,null,2));
    evidence.json
    {
      "source_commit": "2e5589b2180dd432320adda124c4f7bcede119b5",
      "driver_source_unchanged": true,
      "driver_initializations": 1,
      "driver_pid_unchanged": true,
      "fault": {
        "injected_expired_watch_errors": 1,
        "withheld_real_delete_A": true,
        "other_proxy_rejections": 0
      },
      "recovery": {
        "real_list_contains_only": [
          "B"
        ],
        "second_watch_uses_relist_resource_version": true,
        "observed_B_again": true,
        "delivered_delete_A": false
      },
      "event_results": {
        "stale_A_forwarded": true,
        "live_B_forwarded": true,
        "normally_deleted_C_forwarded": false,
        "post_C_live_B_count": 2
      },
      "real_api_checks": {
        "remaining_sandboxes": [
          "B"
        ],
        "all_test_events_exist": true,
        "all_event_references_match_recorded_real_uids": true,
        "workload_pods": 0,
        "pvcs": 0
      },
      "scope": "Unmodified production driver watch entry point against a real kind API; one isolated watch fault was injected. No main gateway/CLI exposure claim."
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions