Skip to content

feat(kubernetes): configure workload and supervisor placement - #5629

Draft
TaylorMutch wants to merge 7 commits into
mainfrom
feat/5619-kubernetes-scheduling/TaylorMutch
Draft

TaylorMutch wants to merge 7 commits into
mainfrom
feat/5619-kubernetes-scheduling/TaylorMutch

Conversation

@TaylorMutch

@TaylorMutch TaylorMutch commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Prefer separate nodes for sandbox workloads and supervisors, including independently deployed shared supervisors. Operators can require separation or opt out, and can optionally select distinct node pools for gateways, supervisors, and workloads.

Related Issue

Closes #5619

Changes

  • Apply the consistent openshell.ai/runtime-role label to workload, supervisor, and gateway pods without changing Service or controller selectors.
  • Add gateway-owned workload placement and role anti-affinity with explicit namespace scope; default to preferred separation across all namespaces.
  • Reconcile placement on stop/start, preserve independently authored affinity and caller settings, normalize Kubernetes toleration defaults, reject conflicting selectors, and validate required placement after admission.
  • Configure reciprocal shared-supervisor rules through Helm, add opt-in supervisor selectors/tolerations, and place PKI hooks with the gateway.
  • Give scheduled pods the configured startup grace after a capacity wait; add one-to-three-node development support, E2E fixtures, and a Kubernetes pod placement guide with scenario diagrams.

Release Notes

Workloads and supervisors prefer separate nodes by default. Node selectors and tolerations remain opt-in. Existing pods acquire role labels and placement rules when recreated.

Testing

  • Checks appropriate to the affected code and behavior pass: driver and E2E Clippy, Rust formatting, Helm lint variants, generated chart docs, rendered-config preflight, resource coherence, ownership/scaling checks, Markdown/Mermaid lint, Fern validation, and docs navigation.
  • Unit tests added/updated: 347 Kubernetes-driver tests and 480 Helm tests pass. Coverage includes policy/scope transitions, placement conflicts, admission validation, and capacity-recovery timing.
  • E2E tests added/updated: sandbox create and stop/start pass with one eligible node using preferred anti-affinity, two eligible nodes using required separation across managed namespaces, and three distinct role pools with tainted supervisor/workload nodes.

The focused live test is mise run e2e:kubernetes:scheduling. It also passes with operator omitted from an operator-configured toleration. Its dedicated Cargo feature excludes the test from the general Kubernetes suite. Test fixtures use matching gateway, sandbox, and supervisor images built from this branch. The isolated validation cluster was removed after testing.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated where applicable: driver README, published gateway configuration, chart docs, and deployment skills.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
…tart

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(kubernetes): prefer workload separation from shared supervisors with common role labels

1 participant