From 72ff348544c1a0dbb0ae14a2dbc135e86db7fe3b Mon Sep 17 00:00:00 2001 From: Matthew Grossman Date: Sat, 21 Nov 2026 12:34:17 -0800 Subject: [PATCH] fix(ci): publish dev Helm charts to the build registry Signed-off-by: Matthew Grossman --- .agents/skills/test-release-canary/SKILL.md | 18 ++++++++++----- .github/actions/release-helm-oci/action.yml | 25 +++++++++++++++++---- .github/workflows/release-canary.yml | 18 ++++++++++++++- .github/workflows/release-dev.yml | 2 ++ skills/debug-openshell-cluster/SKILL.md | 5 +++++ 5 files changed, 58 insertions(+), 10 deletions(-) diff --git a/.agents/skills/test-release-canary/SKILL.md b/.agents/skills/test-release-canary/SKILL.md index 4c6cd19994..e35cb661fe 100644 --- a/.agents/skills/test-release-canary/SKILL.md +++ b/.agents/skills/test-release-canary/SKILL.md @@ -92,8 +92,8 @@ Note `install.sh` is pulled from `raw.githubusercontent.com/ai-openshell/openshe `Release Dev` publishes two chart versions for every dev build (see the push steps in `.github/actions/release-helm-oci/action.yml`): -- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push. -- `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.` — immutable, `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images. +- `oci://ghcr.io/ai-openshell/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push. +- `oci://ghcr.io/ai-openshell/openshell/helm-chart:0.0.0-dev.` — SHA-pinned, with `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images. To smoke-test the chart for a specific dev build, dispatch `Release Dev` on the branch first, then run the kind canary steps locally pointed at the SHA-pinned chart (see "Local kind reproduction" below). The release-canary workflow itself does not currently expose `chart_version` / `image_tag` inputs. @@ -101,12 +101,20 @@ To smoke-test the chart for a specific dev build, dispatch `Release Dev` on the The `kubernetes` job can be reproduced on any machine with Docker and `mise install`-provided `kubectl` + `helm`: +Authenticate Docker and Helm to `ghcr.io` with credentials that have pull access to the dev packages before running these commands. + ```shell kind create cluster --name release-canary-local bash e2e/support/install-agent-sandbox.sh -helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ +for component in gateway sandbox supervisor; do + image="ghcr.io/ai-openshell/openshell/${component}:dev" + docker pull "$image" + kind load docker-image "$image" --name release-canary-local +done + +helm install openshell oci://ghcr.io/ai-openshell/openshell/helm-chart \ --version 0.0.0-dev \ --namespace openshell --create-namespace \ --set server.disableTls=true \ @@ -127,7 +135,7 @@ Keep `pkiInitJob.enabled=true` (the chart default), even when `server.disableTls=true`. The hook also generates the sandbox JWT signing secret that the gateway pod always mounts. -Swap `0.0.0-dev` for `0.0.0-dev.` to pin to a specific dev build. Tear down with `kind delete cluster --name release-canary-local`. +Swap `0.0.0-dev` for `0.0.0-dev.` and `:dev` for `:` to pin to a specific dev build. Tear down with `kind delete cluster --name release-canary-local`. Loopback registration auto-derives the gateway name to `openshell` if `--name` is omitted, which collides with the `install.sh`-installed local gateway — always pass `--name kind` (or another distinct name) when registering in addition to a local install. @@ -142,7 +150,7 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i | `ubuntu-snap-system-docker` fails during the prover checks | The prover artifact is missing or packaged for the wrong architecture, `openshell.prover` is not exposed, missing the alias to `openshell-prover`, or not permitted to read the test policies, or its solver linkage is not runnable. | The `Install and check status` and `Check a policy boundary with the snap prover` steps, plus `snap info openshell` and `snap connections openshell`. | | `ubuntu-snap-docker-preflight` unexpectedly succeeds | The installer no longer fails before installing the OpenShell snap when Docker is absent or supplied by the Docker snap. | Inspect `install.log`, `docker-snap.log`, `snap list`, and snapd changes. | | `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. | -| `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. | +| `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check that `ghcr.io/ai-openshell/openshell/gateway:dev` was loaded into kind. | | `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. | The `kubernetes` job's diagnostics step (only runs `if: failure()`) emits, in order: helm status, rendered manifest, `kubectl get all`, pod descriptions, pod logs (200 lines per container), port-forward log, gateway list, CLI version. Read it top-to-bottom — most failures fall out by the manifest or pod logs. diff --git a/.github/actions/release-helm-oci/action.yml b/.github/actions/release-helm-oci/action.yml index aba7504129..e231280a32 100644 --- a/.github/actions/release-helm-oci/action.yml +++ b/.github/actions/release-helm-oci/action.yml @@ -20,6 +20,14 @@ inputs: release-kind: description: "dev or public — controls the duplicate-version guard." required: true + registry-namespace: + description: "GHCR namespace without the oci:// prefix." + required: false + default: ghcr.io/nvidia/openshell + image-registry: + description: "Override global.image.registry in the packaged gateway chart." + required: false + default: "" pin-sha: description: > Full commit SHA. When set on dev releases, a second chart is also pushed @@ -48,6 +56,7 @@ runs: env: CHART_VERSION: ${{ inputs.chart-version }} APP_VERSION: ${{ inputs.app-version }} + IMAGE_REGISTRY: ${{ inputs.image-registry }} shell: bash run: | set -euo pipefail @@ -59,6 +68,10 @@ runs: sed -i "s/^version:.*/version: ${CHART_VERSION}/" "${chart_dir}/Chart.yaml" sed -i "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" "${chart_dir}/Chart.yaml" done + if [[ -n "${IMAGE_REGISTRY}" ]]; then + sed -i "s|^ registry: ghcr.io/nvidia$| registry: ${IMAGE_REGISTRY}|" "${GATEWAY_CHART_DIR}/values.yaml" + grep -Fxq " registry: ${IMAGE_REGISTRY}" "${GATEWAY_CHART_DIR}/values.yaml" + fi echo "gateway_chart_dir=${GATEWAY_CHART_DIR}" >> "$GITHUB_OUTPUT" echo "workspace_chart_dir=${WORKSPACE_CHART_DIR}" >> "$GITHUB_OUTPUT" echo "chart_version=${CHART_VERSION}" >> "$GITHUB_OUTPUT" @@ -70,17 +83,18 @@ runs: if: inputs.release-kind == 'public' env: CHART_VERSION: ${{ steps.prep.outputs.chart_version }} + REGISTRY_NAMESPACE: ${{ inputs.registry-namespace }} shell: bash run: | set -euo pipefail found=0 for chart in helm-chart openshell-workspace; do - OCI_CHART="oci://ghcr.io/nvidia/openshell/${chart}" + OCI_CHART="oci://${REGISTRY_NAMESPACE}/${chart}" # Only "not found" means unpublished; any other lookup error fails # rather than risk overwriting. if lookup_error=$(helm show chart "${OCI_CHART}" --version "${CHART_VERSION}" 2>&1 >/dev/null); then found=$((found + 1)) - elif [[ "${lookup_error}" != *"ghcr.io/nvidia/openshell/${chart}:${CHART_VERSION}: not found"* ]]; then + elif [[ "${lookup_error}" != *"${REGISTRY_NAMESPACE}/${chart}:${CHART_VERSION}: not found"* ]]; then echo "::error::Could not check whether ${chart}:${CHART_VERSION} is published: ${lookup_error}" exit 1 fi @@ -108,6 +122,8 @@ runs: - name: Push Helm chart to GHCR OCI if: steps.published.outputs.published != 'true' + env: + REGISTRY_NAMESPACE: ${{ inputs.registry-namespace }} shell: bash run: | set -euo pipefail @@ -117,7 +133,7 @@ runs: push_chart() { local attempt for attempt in 1 2 3 4; do - if helm push "$1" oci://ghcr.io/nvidia/openshell; then + if helm push "$1" "oci://${REGISTRY_NAMESPACE}"; then return 0 fi if [ "${attempt}" -lt 4 ]; then @@ -135,6 +151,7 @@ runs: if: inputs.pin-sha != '' env: PIN_SHA: ${{ inputs.pin-sha }} + REGISTRY_NAMESPACE: ${{ inputs.registry-namespace }} GATEWAY_CHART_DIR: ${{ steps.prep.outputs.gateway_chart_dir }} WORKSPACE_CHART_DIR: ${{ steps.prep.outputs.workspace_chart_dir }} shell: bash @@ -144,7 +161,7 @@ runs: push_chart() { local attempt for attempt in 1 2 3 4; do - if helm push "$1" oci://ghcr.io/nvidia/openshell; then + if helm push "$1" "oci://${REGISTRY_NAMESPACE}"; then return 0 fi if [ "${attempt}" -lt 4 ]; then diff --git a/.github/workflows/release-canary.yml b/.github/workflows/release-canary.yml index 7ebbafac6c..a9c8875fd5 100644 --- a/.github/workflows/release-canary.yml +++ b/.github/workflows/release-canary.yml @@ -381,6 +381,9 @@ jobs: if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} runs-on: ubuntu-latest timeout-minutes: 20 + permissions: + contents: read + packages: read env: KIND_CLUSTER_NAME: release-canary-${{ github.run_id }} RELEASE_NAME: openshell @@ -408,10 +411,23 @@ jobs: - name: Install Agent Sandbox controller run: bash e2e/support/install-agent-sandbox.sh + - name: Pull dev runtime images into kind + env: + GHCR_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + printf '%s' "$GHCR_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin + printf '%s' "$GHCR_TOKEN" | helm registry login ghcr.io -u "$GITHUB_ACTOR" --password-stdin + for component in gateway sandbox supervisor; do + image="ghcr.io/ai-openshell/openshell/${component}:dev" + docker pull "$image" + kind load docker-image "$image" --name "$KIND_CLUSTER_NAME" + done + - name: Install OpenShell Helm chart from GHCR OCI run: | set -euo pipefail - helm install "$RELEASE_NAME" oci://ghcr.io/nvidia/openshell/helm-chart \ + helm install "$RELEASE_NAME" oci://ghcr.io/ai-openshell/openshell/helm-chart \ --version 0.0.0-dev \ --namespace "$RELEASE_NAMESPACE" --create-namespace \ --set server.disableTls=true \ diff --git a/.github/workflows/release-dev.yml b/.github/workflows/release-dev.yml index 5900ff2323..21de33825b 100644 --- a/.github/workflows/release-dev.yml +++ b/.github/workflows/release-dev.yml @@ -553,6 +553,8 @@ jobs: chart-version: 0.0.0-dev app-version: dev release-kind: dev + registry-namespace: ghcr.io/ai-openshell/openshell + image-registry: ghcr.io/ai-openshell pin-sha: ${{ github.sha }} publish-fern-docs: diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 2b101ad1e3..43c68dfeeb 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -513,6 +513,11 @@ When `userns` is configured (e.g. `userns = "auto"` or `userns = "keep-id"`): ### Step 6: Check Kubernetes Helm Gateways +For a dev Helm install, verify that the cluster has pull access to the +`ghcr.io/ai-openshell/openshell` runtime images, or that matching images were +preloaded on the nodes. An `ImagePullBackOff` after a successful chart download +can indicate that Helm had registry access but the cluster did not. + ```bash helm -n openshell status openshell helm -n openshell get values openshell