Repository navigation
Description
Update: Complete kind driver-watch relist reproduction and full harness now confirm stale A-event routing after recovery, with live-B and normal-delete-C controls. The production driver source is unchanged.
User Story
I use a kind-based OpenShell Kubernetes demo and inspect sandbox creation and runtime events. I need events to remain associated with the correct sandbox resources after a watch reconnect or resource replacement, so that old Pod events do not mislead diagnosis of a current sandbox.
Problem Statement
The shared-namespace Kubernetes watcher keeps its sandbox-name and Pod-name event indexes for the lifetime of the watch task. It handles InitApply like an ordinary update, but ignores both Init and InitDone. Updates only add entries. Entries are removed only when an explicit Delete arrives.
A kube-runtime relist therefore does not remove entries for resources absent from the new snapshot. This violates the dependency's snapshot contract: after InitDone, objects omitted from the InitApply set should be treated as deleted. A missed delete followed by relist can leave a stale event-correlation entry indefinitely within that watch task.
The event mapper also checks only involved-object kind/name, not involved-object UID. If a retained or reused name belongs to a different Pod, an event can still be attributed through the old association. Snapshot cleanup and UID-aware event attribution are related safeguards; this report does not assume that every old Pod alias is invalid, because delayed historical events may be intentional.
Impact / Why This Matters
A recovered watch can keep obsolete event routes and accumulate stale index entries. If an event names an obsolete or reused object, it can be routed to a sandbox stream using an association that no longer matches the Kubernetes object.
The demonstrated effect is event-index and attribution correctness. This report does not claim a workload isolation bypass, routine cross-tenant disclosure, or permanent gateway sandbox-record loss. The gateway has a separate reconciliation sweep for missing backend resources and closes event buffers for cleaned-up IDs.
Acceptance Criteria
- After a completed relist, the active event indexes agree with the authoritative snapshot, including resources absent from that snapshot.
- Events cannot be attributed to a live sandbox merely because a different Kubernetes object reused an indexed name.
- Explicit deletion still removes the corresponding index entries.
- An interrupted relist does not publish a partial index as a complete authoritative snapshot.
- Tests cover an omitted resource after relist, resource replacement, delayed old-UID events, and normal current-UID events.
- The intended retention policy for historical Pod aliases is explicit and covered by tests.
Reproduction Steps
Use an OpenShell deployment with the Kubernetes driver in shared workspace mode and an expendable sandbox.
- Create sandbox A and allow its Sandbox and Pod names to enter the driver's event indexes.
- During an interruption of the driver's Kubernetes watch, remove A. Allow the prior watch resourceVersion to expire so recovery performs a fresh list rather than resuming the old watch history.
- Restore watch access. The new list completes without A. The current code processes
Init, the surviving objects'InitApplyevents, andInitDonewithout replacing or pruning its event indexes. - A Kubernetes event naming A's old CR or Pod still resolves through the stale entry. A name-reuse test should use a different involved-object UID and verify that it is not attributed using the old association.
An ordinary brief disconnect may resume the previous watch rather than relist. Restarting the entire gateway also clears the maps and does not reproduce the retained-map case. Use an isolated test deployment when controlling the watch interruption; do not disrupt unrelated sandbox sessions.
A complete isolated driver-watch relist sequence has now been exercised against a real kind API: withheld DELETE, one injected 410, a real post-fault LIST containing only B, and a replacement watch using that list's resourceVersion. See the complete reproduction, controls, and harness.
Deterministic Verification
A temporary test compiled the current update_indexes, remove_indexes, and map_kube_event_to_platform helpers with lightweight data/timestamp fixtures. It exercised the same additive updates used by the current watch arms.
Apply A -> Init -> InitApply B -> InitDone
Result: A's resource-name and Pod-name entries remain.
Old sandbox ID -> reused Pod name
Pod event: same name, explicitly different UID
Result: event maps to the old sandbox ID.
Explicit Delete A
Result: all indexed aliases for A are removed.
Four helper behavior checks passed. This verifies the association behavior, not the complete async watch or a normal cross-sandbox exposure path.
kind baseline (initial verification)
A disposable sandbox was created with OpenShell 0.1.8 in shared workspace mode. A clearly marked synthetic Kubernetes Event referenced that sandbox's current workload Pod UID. A direct WatchSandbox subscription with follow_events=true and event_tail=200 received exactly that test marker from source kubernetes.
The sandbox was subsequently deleted, and the test Event still existed with the original Pod UID. This establishes a working deployment-level event-observation path and a retained old-object Event for the next test. That was only the initial event-observation baseline. The subsequent complete driver-watch relist test now confirms the Sandbox-name index defect using Suspended CR fixtures. Its test Events reference real Sandbox CR UIDs; it does not claim an additional live Pod-runtime or Gateway/CLI exposure result.
No third-party packages were installed: the API observer used the existing Node.js runtime's built-in HTTP/2 and TLS modules. Published evidence omits hostnames, internal addresses, private paths, and client credentials.
Environment
- Source checked:
2e5589b2180dd432320adda124c4f7bcede119b5. - The same no-op relist arms and index helpers are present in
v0.1.8. - kube-runtime: 0.99.0.
- kind baseline: OpenShell CLI/gateway 0.1.8; Agent Sandbox controller v1.0.3; kind v0.33.0; Kubernetes v1.37.0; Linux x86_64.
- Workspace mode: shared.
Code References
- Watch-task lifetime indexes and additive InitApply handling
- Relist boundary events are ignored
- Index update and explicit deletion helpers
- Name-only event association
- kube-runtime Event snapshot contract
This is separate from #5028: that issue concerns missing Event forwarding in managed/operator mode, while this report concerns the indexes in the existing shared-mode Event forwarding path.
Activity
cloudonly commentedon Nov 16, 2026
Complete driver-watch relist reproduction on kind
Confirmed against a real kind API with the unmodified production
openshell-driver-kuberneteslibrary at2e5589b2180dd432320adda124c4f7bcede119b5. The probe links that library and calls its publicwatch_sandboxes()entry point. It does not copy or substitute the watch loop, relist handling, readiness conversion, index functions, or event mapper.Exact scope
ERRORframe withcode=410,reason=Expired. The subsequent recovery uses the actual kube-runtime state machine and actual API responses.Environment: kind v0.33.0; Kubernetes v1.37.0; Agent Sandbox controller v1.0.3; kube-runtime 0.99.0. The existing Gateway is 0.1.8, but the producer under test is the linked driver library at the pinned source commit above.
Observed sequence
DELETED Anotification and withholds it; the driver emits noDeleted(A).InitDonepath. The harness does not fabricateInit,InitApply, orInitDoneevents.The late A Event leaves the original driver as source
kubernetes, reasonReproAfterRelistA, sandbox IDfixture-a, even though A was omitted from the completed real list. C supplies the comparison: the normal DELETE branch removes that route. All three Event objects were independently checked in the actual API, including their references to the real captured UIDs; only B remained among the Sandbox CRs.Executed assertions
The verifier passed all of these checks:
{ "driver_source_unchanged": true, "driver_initializations": 1, "driver_pid_unchanged": true, "injected_expired_watch_errors": 1, "withheld_real_delete_A": true, "other_proxy_rejections": 0, "real_relist_contains_only": ["B"], "second_watch_uses_relist_resource_version": true, "observed_B_again": true, "delivered_delete_A": false, "stale_A_forwarded": true, "live_B_forwarded": true, "normally_deleted_C_forwarded": false, "post_C_live_B_count": 2, "all_test_events_exist_in_real_api": true, "all_event_references_match_recorded_real_uids": true, "workload_pods": 0, "pvcs": 0 }This closes the earlier gap between helper-only verification and a complete asynchronous driver-watch recovery test. It also keeps the attribution claim narrow: a stale driver route is demonstrated, while downstream cleanup/tombstone behavior remains a separate boundary.
The isolated namespace and test Events were removed after the assertions. The observer, fault proxy, API proxy, and experiment tunnel were stopped. The original Gateway and existing sandbox remained Ready. Runtime PIDs, actual resource UIDs/resourceVersions, SSH aliases, host addresses, private paths, and client credentials are omitted from this report.
Reproduction harness
The expandable sections below contain the complete harness and step-by-step procedure. Save them under
demo/issue-repros/5596/in a checkout of the pinned source. Use an existing configured contributor toolchain, Node.js, kubectl, and a test kind cluster with the Agent Sandbox CRD. No additional packages need to be installed solely for this test.README.md
probe.rs
build-probe.mjs
fault-proxy.mjs
namespace.yaml
fixtures.yaml
control-c.yaml
make-events.mjs
verify-run.mjs
evidence.json
{ "source_commit": "2e5589b2180dd432320adda124c4f7bcede119b5", "driver_source_unchanged": true, "driver_initializations": 1, "driver_pid_unchanged": true, "fault": { "injected_expired_watch_errors": 1, "withheld_real_delete_A": true, "other_proxy_rejections": 0 }, "recovery": { "real_list_contains_only": [ "B" ], "second_watch_uses_relist_resource_version": true, "observed_B_again": true, "delivered_delete_A": false }, "event_results": { "stale_A_forwarded": true, "live_B_forwarded": true, "normally_deleted_C_forwarded": false, "post_C_live_B_count": 2 }, "real_api_checks": { "remaining_sandboxes": [ "B" ], "all_test_events_exist": true, "all_event_references_match_recorded_real_uids": true, "workload_pods": 0, "pvcs": 0 }, "scope": "Unmodified production driver watch entry point against a real kind API; one isolated watch fault was injected. No main gateway/CLI exposure claim." }