Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions deploy/helm/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -577,6 +577,7 @@ discovery endpoint or its TLS CA.
| gateway.image.registry | string | `""` | Gateway image registry. Empty uses global.image.registry. |
| gateway.image.repository | string | `"openshell/gateway"` | Gateway image repository. |
| gateway.image.tag | string | `""` | Gateway image tag. Defaults to the chart appVersion when empty. |
| gateway.listenAddress | string | `"0.0.0.0"` | Gateway listener IP. Use `::` for IPv6. |
| gatewayConfig | object | `{"openshell":{"version":2}}` | Non-secret gateway application configuration. Top-level keys name TOML tables and are rendered into the mounted gateway.toml file. Kubernetes resource inputs remain outside this map; template expressions derive the corresponding runtime values from their resource owner. |
| global.image.pullPolicy | string | `"IfNotPresent"` | Shared OpenShell image pull policy. Individual image pull policies take precedence. |
| global.image.registry | string | `"ghcr.io/nvidia"` | Shared OpenShell image registry. Individual image registries take precedence. |
Expand Down Expand Up @@ -756,6 +757,7 @@ discovery endpoint or its TLS CA.
| supervisor.image.registry | string | `""` | Supervisor image registry. Empty uses global.image.registry. |
| supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. |
| supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. |
| supervisor.listenAddress | string | `"0.0.0.0"` | Shared supervisor listener IP. Use `::` for IPv6. |
| supervisor.network.additionalCaConfigMapName | string | `""` | Name of an operator-owned ConfigMap with a `ca.crt` key containing additional destination CA certificates for sandbox supervisors. Empty disables additional destination trust. |
| supervisor.podDisruptionBudget.enabled | bool | `true` | Render a PodDisruptionBudget with maxUnavailable 1 for the shared supervisor, used only when gatewayConfig selects supervisor_topology = "gateway". Disable it only when maintenance must evict more than one supervisor Pod at a time, or evict a member while another is unavailable. |
| supervisor.podSecurityContext.fsGroup | int | `65534` | fsGroup assigned to the shared supervisor pod. |
Expand Down
4 changes: 4 additions & 0 deletions deploy/helm/openshell/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -596,6 +596,10 @@ default to enabled; an explicit false must not be replaced by that default.
{{- if $enabled -}}true{{- end -}}
{{- end }}

{{- define "openshell.listenHost" -}}
{{- if contains ":" . -}}[{{ . }}]{{- else -}}{{ . }}{{- end -}}
{{- end }}

{{/*
Validate a non-empty, user-provided Kubernetes Secret name. Secret data never
passes through Helm values into gateway.toml; only this reference is rendered.
Expand Down
7 changes: 4 additions & 3 deletions deploy/helm/openshell/templates/_toml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,10 @@ Top-level lists represent TOML arrays of tables and preserve their YAML order. *
{{- $gateway := get $config "openshell.gateway" | default dict -}}
{{- if not (hasKey $gateway "name") -}}{{- $_ := set $gateway "name" (get $legacyServer "name" | default (include "openshell.fullname" .)) -}}{{- end -}}
{{/* The Service and probes own listener ports, so keep runtime listeners aligned. */}}
{{- $_ := set $gateway "bind_address" (printf "0.0.0.0:%v" .Values.service.port) -}}
{{- if .Values.service.healthPort -}}{{- $_ := set $gateway "health_bind_address" (printf "0.0.0.0:%v" .Values.service.healthPort) -}}{{- else -}}{{- $_ := unset $gateway "health_bind_address" -}}{{- end -}}
{{- if include "openshell.metricsEnabled" . -}}{{- $_ := set $gateway "metrics_bind_address" (printf "0.0.0.0:%v" .Values.service.metricsPort) -}}{{- else -}}{{- $_ := unset $gateway "metrics_bind_address" -}}{{- end -}}
{{- $listenHost := include "openshell.listenHost" .Values.gateway.listenAddress -}}
{{- $_ := set $gateway "bind_address" (printf "%s:%v" $listenHost .Values.service.port) -}}
{{- if .Values.service.healthPort -}}{{- $_ := set $gateway "health_bind_address" (printf "%s:%v" $listenHost .Values.service.healthPort) -}}{{- else -}}{{- $_ := unset $gateway "health_bind_address" -}}{{- end -}}
{{- if include "openshell.metricsEnabled" . -}}{{- $_ := set $gateway "metrics_bind_address" (printf "%s:%v" $listenHost .Values.service.metricsPort) -}}{{- else -}}{{- $_ := unset $gateway "metrics_bind_address" -}}{{- end -}}
{{- range $legacyKey, $runtimeKey := dict "logLevel" "log_level" "enableLoopbackServiceHttp" "enable_loopback_service_http" "enableWebsocketTunnel" "enable_websocket_tunnel" "policyValidationFailureMode" "policy_validation_failure_mode" -}}
{{- if not (hasKey $gateway $runtimeKey) -}}{{- $_ := set $gateway $runtimeKey (get $legacyServer $legacyKey) -}}{{- end -}}
{{- end -}}
Expand Down
7 changes: 4 additions & 3 deletions deploy/helm/openshell/templates/supervisor.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
{{- $app := include "openshell.appListenerConfig" . | fromYaml }}
{{- $appDeployment := include "openshell.appListenerDeployment" . | fromYaml }}
{{- $appPort := get $app "port" | default 8443 }}
{{- $listenHost := include "openshell.listenHost" .Values.supervisor.listenAddress }}
{{- if get $app "enabled" }}
{{- $_ := required "supervisor app listener requires domain" (get $app "domain") }}
{{- $_ := required "supervisor app listener requires tls_secret_name" (get $appDeployment "tls_secret_name") }}
Expand Down Expand Up @@ -88,19 +89,19 @@ spec:
- --gateway-endpoint
- {{ $endpoint | quote }}
- --address
- 0.0.0.0:5501
- {{ printf "%s:5501" $listenHost | quote }}
- --token-file
- /etc/openshell/supervisor-token/token
- --runtime-listen
- 0.0.0.0:{{ get $shared "runtime_port" }}
- {{ printf "%s:%v" $listenHost (get $shared "runtime_port") | quote }}
- --session-root
- /run/openshell/sessions
env:
{{- if get $app "enabled" }}
- name: OPENSHELL_APP_LISTENER_ENABLED
value: "true"
- name: OPENSHELL_APP_LISTEN_ADDRESS
value: {{ printf "0.0.0.0:%v" $appPort | quote }}
value: {{ printf "%s:%v" $listenHost $appPort | quote }}
- name: OPENSHELL_APP_SERVICE_DOMAIN
value: {{ get $app "domain" | quote }}
- name: OPENSHELL_APP_TLS_CERT
Expand Down
14 changes: 14 additions & 0 deletions deploy/helm/openshell/tests/gateway_config_serializer_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,20 @@ tests:
path: data["gateway.toml"]
pattern: '(?m)^metrics_bind_address = "0\.0\.0\.0:19090"$'

- it: brackets an IPv6 listen address
set:
gateway.listenAddress: "::"
asserts:
- matchRegex:
path: data["gateway.toml"]
pattern: '(?m)^bind_address = "\[::\]:8080"$'
- matchRegex:
path: data["gateway.toml"]
pattern: '(?m)^health_bind_address = "\[::\]:8081"$'
- matchRegex:
path: data["gateway.toml"]
pattern: '(?m)^metrics_bind_address = "\[::\]:9090"$'

- it: aligns runtime namespace and ServiceAccount with chart-owned resources
set:
server.sandboxNamespace: chart-sandboxes
Expand Down
16 changes: 16 additions & 0 deletions deploy/helm/openshell/tests/supervisor_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,22 @@ tests:
- hasDocuments:
count: 0
template: templates/supervisor-pdb.yaml
- it: brackets an IPv6 listen address
template: templates/supervisor.yaml
set:
supervisor.listenAddress: "::"
gatewayConfig:
openshell.drivers.kubernetes:
supervisor_topology: gateway
asserts:
- equal:
path: spec.template.spec.containers[0].args[4]
value: "[::]:5501"
documentIndex: 0
- equal:
path: spec.template.spec.containers[0].args[8]
value: "[::]:5500"
documentIndex: 0
- it: deploys the supervisor pool without Kubernetes identity in the gateway namespace
template: templates/supervisor.yaml
set:
Expand Down
4 changes: 4 additions & 0 deletions deploy/helm/openshell/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ gateway:
tag: ""
# -- Gateway image digest. When set, this takes precedence over tag.
digest: ""
# -- Gateway listener IP. Use `::` for IPv6.
listenAddress: 0.0.0.0

# Trusted workload-side runtime image.
sandboxRuntime:
Expand Down Expand Up @@ -69,6 +71,8 @@ supervisor:
tag: ""
# -- Supervisor image digest. When set, this takes precedence over tag.
digest: ""
# -- Shared supervisor listener IP. Use `::` for IPv6.
listenAddress: 0.0.0.0
# -- Shared supervisor pool members, used only when gatewayConfig selects
# supervisor_topology = "gateway". The gateway always coordinates the shared
# supervisor as a high-availability pool: when a member fails or leaves,
Expand Down
2 changes: 2 additions & 0 deletions docs/how-it-works/gateways/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1373,6 +1373,8 @@ Kubernetes configurations set `namespace`, `service_account_name`, and `enable_u

The gateway runs as a Pod and creates paired workload and supervisor Pods in another namespace. The gateway CA is projected from a Kubernetes Secret into supervisor Pods; user client certificate and key entries in that Secret are not exposed to either Pod.

On IPv6 clusters, set the Helm values `gateway.listenAddress` and `supervisor.listenAddress` to `::`.

```toml
[openshell]
version = 2
Expand Down
1 change: 1 addition & 0 deletions skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -1608,6 +1608,7 @@ credential failures.
| Symptom | Likely cause | Check |
|---|---|---|
| `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs |
| Probes get `connection refused` on an IPv6 cluster | Listeners bind `0.0.0.0` by default | Set `gateway.listenAddress=::` and `supervisor.listenAddress=::` |
| Collector receives no OCSF log records, or `openshell.ocsf.gap` records | `[openshell.gateway.otlp.ocsf]` missing or without `enabled = true`, an unreachable endpoint or one that does not speak the table's `protocol`, or export failures and queue pressure | Check `[openshell.gateway.otlp.ocsf]` for `enabled = true` and the endpoint it resolves to, the gateway's `OTLP exporting enabled` startup log for `ocsf_endpoint` and `ocsf_protocol`, export warnings, and `openshell_ocsf_otlp_*` metrics. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). |
| Gateway OCSF JSONL stops growing, has gaps, or is rejected by a SIEM | File errors, queue pressure, or shipper falling behind retention | Inspect gateway warnings and `openshell_ocsf_log_*` metrics; check `[openshell.gateway.ocsf_log]`, directory permissions, free space, per-replica paths, and shipper rotation checkpoints. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). |
| `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with `OPENSHELL_DEV_OTLP_ENDPOINT` set but no collector is listening, for example because `helm:k3s:forward` stopped | Start the collector forwarding again, or restart the gateway task without `OPENSHELL_DEV_OTLP_ENDPOINT`; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` |
Expand Down
Loading