Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 45 additions & 2 deletions .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ Port mappings created at cluster time (cannot be changed without recreating):
| `8080` | Port `80` via k3d load balancer | Envoy Gateway LoadBalancer service (`values-gateway.yaml`) |

Override with env vars before running `helm:k3s:create`:
- `HELM_K3S_NODES` (default: `1`; accepts `1`, `2`, or `3` total nodes for a new cluster)
- `HELM_K3S_LB_HOST_PORT` (default: `8080`)
- `HELM_K3S_PRELOAD_SANDBOX_IMAGE` (default:
`nvcr.io/nvidia/base/ubuntu:24.04`; set to an empty value to skip)
Expand Down Expand Up @@ -125,8 +126,8 @@ chart renders the gateway's
`recovery_timeout_secs` from `supervisor.recoveryTimeoutSeconds`, default 60;
the driver's `shared_supervisor` table stays unchanged), a
RollingUpdate Deployment (`maxSurge: 1`, `maxUnavailable: 0`) with a 60-second
grace period, no affinity unless `supervisor.affinity` is set (top-level
`affinity` applies only to the gateway), the pool identity env vars
grace period, preferred role anti-affinity merged with `supervisor.affinity`
(top-level `affinity` applies only to the gateway), the pool identity env vars
(`OPENSHELL_SUPERVISOR_POOL_MEMBER_ID` and `_ADVERTISE_IP` from the Pod name
and IP), and a `maxUnavailable: 1` PodDisruptionBudget unless
`supervisor.podDisruptionBudget.enabled=false` (default `true`; toggling it
Expand Down Expand Up @@ -273,6 +274,48 @@ registers the gateway under the worktree-specific k3d
cluster name and selects it as the active gateway. Keep the forwarding
task running while using those endpoints.

### Role scheduling validation

See the published `/kubernetes/pod-placement` guide for deployment examples and role diagrams.

Workloads, supervisors, and gateways use `openshell.ai/runtime-role` with values
`workload`, `supervisor`, and `gateway`. Workload/supervisor anti-affinity defaults
to preferred and matches across all namespaces. Set
`gatewayConfig.openshell.drivers.kubernetes.sandbox_runtime.pod_anti_affinity`
to `required` for hard separation or `off` to opt out. The driver's
`pod_anti_affinity_namespace_selector` narrows namespace scope. The chart derives
shared-supervisor rules from the same gateway configuration.
Node selectors and tolerations remain opt-in. Gateways use the top-level chart
fields; supervisors use `supervisor.nodeSelector` and `supervisor.tolerations`;
workloads use gateway TOML `sandbox_runtime.workload_placement`.

Create an isolated cluster with up to three nodes:

```bash
HELM_K3S_CLUSTER_NAME=openshell-scheduling HELM_K3S_NODES=3 \
HELM_K3S_LB_HOST_PORT=18080 mise run helm:k3s:create
```

`HELM_K3S_NODES` affects new clusters only; it does not resize an existing one.
Use `mise run e2e:kubernetes:scheduling` for the shared-supervisor create/restart
checks. This task enables the dedicated `e2e-kubernetes-scheduling` feature;
the scheduling test is excluded from the general Kubernetes suite. Against the
isolated cluster, set `OPENSHELL_E2E_KUBE_CONTEXT` and
`OPENSHELL_E2E_KUBE_BUILD_IMAGES=1` to build matching branch images.
Layer `ci/values-scheduling-required.yaml` through
`OPENSHELL_E2E_KUBE_EXTRA_VALUES` and set `OPENSHELL_E2E_SCHEDULING_MODE=required`
for hard separation. The `ci/values-scheduling-pools.yaml` fixture additionally
requires one node labeled `openshell.ai/node-pool=<role>` for each of
`gateway`, `supervisor`, and `workload`. Taint the two worker pools with
`openshell.ai/node-pool=<role>:NoSchedule` to exercise tolerations; leave the
gateway node untainted for the local-path storage provisioner
(or configure that provisioner to tolerate the gateway taint). Set
`OPENSHELL_E2E_SCHEDULING_POOLS=1` to verify their nodes. Add the managed-workspace
overlay to check cross-namespace placement. Verify a one-eligible-node preferred
case as well. Do not cordon, relabel, or taint nodes in another development
cluster. A pending required workload gets fresh bounded startup grace when
capacity returns; it must not be reaped immediately after becoming scheduled.

### Viewing local traces

The gateway exports OTLP/gRPC to
Expand Down
27 changes: 27 additions & 0 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,33 @@ The supervisor Pod has a direct, non-controller owner reference to the Sandbox
resource. This links its garbage-collection lifecycle to the sandbox without
competing with the Agent Sandbox controller for workload-Pod ownership.

### Scheduling

The operator-owned `sandbox_runtime.pod_anti_affinity` defaults to `preferred`;
`required` enforces workload/supervisor node separation and `off` removes the
generated rule. The driver uses the reserved `openshell.ai/runtime-role` label
(`workload` or `supervisor`), hostname topology, and a native
`pod_anti_affinity_namespace_selector` (empty means all namespaces). Helm also
labels gateways and shared supervisors and renders reciprocal supervisor rules.

Optional `sandbox_runtime.workload_placement.node_selector` and `tolerations`
apply independently of caller driver-config opt-in. Conflicting caller selectors
fail; tolerations append without duplicates. A reserved template annotation
records operator additions so stop/start can replace them while preserving
caller placement and unrelated admission rules. Restart reconciles only the
exact anti-affinity term recorded as inserted by the driver, using a merge patch
that clears removed keys. Independently authored role rules retain their scope
and strength.
Admission checks validate required workload placement before releasing the
bootstrap scheduling gate. A verified unschedulable generation may wait for
capacity; once bound, its scheduling transition starts a bounded startup grace.

Standalone drivers expose `--pod-anti-affinity`,
`--pod-anti-affinity-namespace-selector` (JSON), and `--workload-placement` (JSON),
with corresponding `OPENSHELL_K8S_*` environment variables. See the
[pod placement guide](../../docs/kubernetes/pod-placement.mdx)
for deployment examples and cross-namespace scope.

When the gateway exports OTLP traces, the driver sets
`OPENSHELL_OTLP_ENDPOINT` on the supervisor Pod to the gateway's endpoint and
`TRACEPARENT` to the trace context of the operation that created the Pod. The
Expand Down
28 changes: 28 additions & 0 deletions crates/openshell-driver-kubernetes/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,12 @@ impl FromStr for KubernetesControlConnection {
#[serde(default, deny_unknown_fields)]
pub struct KubernetesSandboxRuntimeConfig {
pub app_listener: KubernetesAppListenerConfig,
/// Separation of workload pods from supervisor pods (preferred by default).
pub pod_anti_affinity: crate::scheduling::KubernetesPodAntiAffinity,
/// Namespaces whose opposite-role pods participate; empty selects all.
pub pod_anti_affinity_namespace_selector:
k8s_openapi::apimachinery::pkg::apis::meta::v1::LabelSelector,
pub workload_placement: crate::scheduling::KubernetesWorkloadPlacement,
/// TCP port of the Sandbox Protocol listener: the workload boundary for
/// supervisor-initiated sandboxes, the supervisor for runtime-initiated ones.
pub boundary_port: u16,
Expand All @@ -129,6 +135,10 @@ impl Default for KubernetesSandboxRuntimeConfig {
Self {
boundary_port: 5500,
app_listener: KubernetesAppListenerConfig::default(),
pod_anti_affinity: crate::KubernetesPodAntiAffinity::default(),
pod_anti_affinity_namespace_selector:
k8s_openapi::apimachinery::pkg::apis::meta::v1::LabelSelector::default(),
workload_placement: crate::KubernetesWorkloadPlacement::default(),
}
}
}
Expand All @@ -146,6 +156,24 @@ pub struct KubernetesAppListenerConfig {
impl KubernetesSandboxRuntimeConfig {
pub fn validate(&self) -> Result<(), String> {
self.app_listener.common.validate()?;
for toleration in &self.workload_placement.tolerations {
let operator = toleration.operator.as_deref().unwrap_or("Equal");
if !matches!(operator, "Equal" | "Exists")
|| (operator == "Exists"
&& toleration
.value
.as_ref()
.is_some_and(|value| !value.is_empty()))
|| (operator == "Equal" && toleration.key.as_ref().is_none_or(String::is_empty))
|| toleration.effect.as_deref().is_some_and(|effect| {
!matches!(effect, "" | "NoSchedule" | "PreferNoSchedule" | "NoExecute")
})
|| (toleration.toleration_seconds.is_some()
&& toleration.effect.as_deref() != Some("NoExecute"))
{
return Err("invalid sandbox_runtime.workload_placement.tolerations: use Kubernetes toleration operators and effects".into());
}
}
if self.app_listener.common.enabled {
if !is_dns_subdomain(&self.app_listener.tls_secret_name) {
return Err("sandbox_runtime.app_listener.tls_secret_name is required when the app listener is enabled".into());
Expand Down
Loading
Loading