Repository navigation
fix(airgap): disconnected deployment support and bootstrap fixes - #164
Merged
butler54 merged 9 commits intoNov 2, 2026
Merged
Conversation
…or 1.2.1 The imageset pinned OCP 4.22.8 while the deployed installer is 4.22.17, and trustee-operator 1.2.0 while values-baremetal pins CSV v1.2.1. The mirror must contain the exact release the installer consumes (K12) and the operator bundles the pattern resolves. Version audit: specs/002-multi-tranche-testing/version-audit.md (coco-gsd)
… in imageset oc-mirror v2 fails catalog filtering when the package's catalog default channel is filtered out: openshift-gitops-operator defaults to 'latest' (we pin gitops-1.22) and gpu-operator-certified defaults to v26.7 (we pin v26.3). The defaultChannel field re-points the default to the pinned channel. Known-failure class recorded as K13 in the T1 mirror checklist (coco-gsd specs/002-multi-tranche-testing/mirror-known-failures.md).
…t is a hard mirror error
registry.redhat.io rejects :latest for container-native-virtualization/virtio-win
('must be pulled with an explicit version'), which fails oc-mirror on every run.
The three guest images are documented-optional (~2 GB each) and unconsumed:
Phase 26 sets enableCommonBootImageImport: false, so CNV never imports them.
Commenting them out also removes the large-blob timeout flakiness (rhel8/rhel9
guest images hit context-deadline-exceeded on upload).
T1 mirror run evidence: coco-gsd specs/002-multi-tranche-testing/t1-progress.md
…disabled subscription The gpu-operator subscription is commented out in values-baremetal.yaml (NVIDIA bundle mirror issues); nothing in T1 consumes the bundle. The nvcr.io related images (dcgm, driver) hit repeated copy timeouts across two resumable retries, blocking mirror certification for content the disconnected deployment never pulls. Tranche 2 re-enables the GPU operator at v26.7 (pawelpros cherry-pick), which mandates its own re-mirror + re-certification per the T022 trigger logic — at that point the bundle returns to the imageset.
The mirror registry is now identified by its DNS name everywhere: - Phase 0-0: canonical MIRROR_REGISTRY=user-jump.int.apac-tech-lab.net:8443, with the resolved IP recorded as a generation-time fact only - Phase 0-1b: cert CN/SAN carry the DNS name (IP SAN kept for transition) - Phase 0-1h: certs.d trust entries for both name and IP - oc-mirror targets the DNS name so generated cluster-resources (IDMS/ITMS/ CatalogSources) reference it — no IP enters the ignition chain - itms-manual-mirrors placeholders substituted from MIRROR_REGISTRY at apply time (was applied verbatim with the literal placeholder) - git-server URLs and repoURL hints use the DNS name Lab IPs are not stable (the jump host moved subnets on 2026-09-15, breaking the T013 install through the reserved-VLAN routing); DNS is maintained through moves and the node's ignition resolver uses it. Spec: FR-017 (specs/002-multi-tranche-testing, R6)
…crets.sh The script only uncommented registryCaCert and bootstrap_secrets. The branch-pair contract for disconnected testing also requires tdxCollateral enabled (offline DCAP collateral) and intelPcsApiKey disabled (manual PCK provisioning via dcap-pck.py in airgap; the connected key file must not be required by load-secrets). - Wrap the tdxCollateral block in AIRGAP-TDX-COLLATERAL markers in values-secret.yaml.template (gen-secrets copies the template verbatim, so markers flow through to the generated file) - Uncomment both marker blocks; comment out the intelPcsApiKey entry - Idempotent; verified against a fresh template copy (YAML-valid output, airgap secrets active, intelPcsApiKey disabled) Contract: specs/002-multi-tranche-testing/contracts/branch-pair.md (coco-gsd)
…64 hex
collect-snp-vcek-urls.sh extracted the hardware ID with
`grep -oP '[0-9a-fA-F]{64}'`, truncating the chip_id to the first 32
bytes. The SNP verifier derives the on-disk VCEK directory from the full
hex(chip_id) — 128-hex for non-Turin EPYC, 16-hex for Turin — so a
truncated value can never match and attestation fails with
POST /kbs/v0/attest returning 401, leaving workloads in
CreateContainerError.
Extract the full chip_id as the KDS URL's trailing path segment (query
stripped) without truncating.
Refs validatedpatterns/trustee-chart#45.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
butler54
commented
Nov 2, 2026
butler54
left a comment
Collaborator
Author
There was a problem hiding this comment.
The changes were slimmed too far
| # Connected mode: repoURL is inferred from your local git remote. | ||
| # Disconnected: uncomment and set to the HTTP URL served by git-http-server.py | ||
| # repoURL: http://JUMP_HOST_IP:8080/coco-pattern.git | ||
| # repoURL: http://user-jump.int.apac-tech-lab.net:8080/coco-pattern.git |
Collaborator
Author
There was a problem hiding this comment.
This is an internal URL don't expose it.
Comment on lines
+162
to
+171
|
|
||
| .PHONY: argocd-login | ||
| argocd-login: ## Extract ArgoCD credentials from cluster and log in with argocd CLI (diagnostic) | ||
| @ARGOCD_NS=$$(oc get argocd -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null) && \ | ||
| ARGOCD_NAME=$$(oc get argocd -n $$ARGOCD_NS -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) && \ | ||
| ARGOCD_ROUTE=$$(oc get route $${ARGOCD_NAME}-server -n $$ARGOCD_NS -o jsonpath='{.spec.host}' 2>/dev/null) && \ | ||
| ARGOCD_PASS=$$(oc get secret $${ARGOCD_NAME}-cluster -n $$ARGOCD_NS -o jsonpath='{.data.admin\.password}' 2>/dev/null | base64 -d) && \ | ||
| echo "ArgoCD URL: https://$$ARGOCD_ROUTE" && \ | ||
| echo "Username: admin" && \ | ||
| echo "Password: $$ARGOCD_PASS" |
Collaborator
Author
There was a problem hiding this comment.
This has not actually logged into argocd. Ensure it logs in.
| ARGOCD_CLI_DIR ?= $(HOME)/.local/bin | ||
|
|
||
| .PHONY: argocd-install | ||
| argocd-install: ## Download argocd CLI from the cluster's ArgoCD image into ~/.local/bin |
Collaborator
Author
There was a problem hiding this comment.
Add this utiluty back in
| @scripts/airgap-post-install.sh --fix-manifest-lists | ||
|
|
||
| .PHONY: airgap-sync-repos | ||
| airgap-sync-repos: ## Push working copy changes to bare HTTP repos and restart git server |
| OC_MIRROR_WORKSPACE ?= file://$(HOME)/oc-mirror-workspace | ||
|
|
||
| .PHONY: airgap-mirror | ||
| airgap-mirror: ## Mirror content to disconnected registry (requires MIRROR_REGISTRY) |
Collaborator
Author
There was a problem hiding this comment.
Isn't this just a wrapper of the mirroring process? isn't it still useful?
| @scripts/collect_firmware_refvals.py --platform baremetal | ||
|
|
||
| .PHONY: collect-azure-refvals | ||
| collect-azure-refvals: ## Collect PCR reference values (Azure) |
Collaborator
Author
There was a problem hiding this comment.
I think this is still required.
butler54
force-pushed
the
feat/airgap-disconnected-support
branch
3 times, most recently
from
November 2, 2026 02:13
29a997c to
95a98e6
Compare
butler54
force-pushed
the
feat/airgap-disconnected-support
branch
4 times, most recently
from
November 2, 2026 02:33
a1d012c to
a8db4eb
Compare
…unbook Review feedback on the PR: - argocd-login now actually logs in via the argocd CLI when available (falls back to printing credentials with a hint to install the CLI) - Scrubbed a leaked internal URL from a commented example in values-global.yaml - Added the rewritten deploy runbook (Quay primary, Docker registry optional appendix) - All 31 Makefile targets retained (multi-platform: Intel TDX, AMD SNP, Azure) - Archived superseded documentation in deprecated/ directories No targets were removed — the earlier retirement was too aggressive.
butler54
force-pushed
the
feat/airgap-disconnected-support
branch
from
November 2, 2026 02:50
a8db4eb to
1b3d31a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR: Airgap/Disconnected Deployment Support
Summary
This PR adds comprehensive disconnected (airgapped) deployment support to the
CoCo pattern, validated end-to-end on a real bare-metal TDX cluster with two
consecutive clean-slate deployments against a Quay registry as the sole
mirror target.
What's included
Imageset correctness (4 commits):
defaultChanneloverrides for operators whose default channels arefiltered out by oc-mirror (gitops
latest, gpu-operatorv26.7)virtio-win:latestis a hard mirror error)pattern-ui-catalog(hybrid OCI-index manifest rejected by Quay;nothing in the pattern or operator CSV deploys it)
range-pin aging: values use
~0.1.10which resolves 0.1.11)Airgap bootstrap (3 commits):
enable-airgap-secrets.sh(handlestdxCollateralandintelPcsApiKeymarkers, not just registryCaCert andbootstrap_secrets)
enable-airgap-secretsoperates at the values-secret convention location(
~/values-secret-<pattern>.yaml— wheregen-secretswrites and theAnsible loader reads)
Operational fixes (2 commits):
chip_idinstead of truncating to 64 hexreg.kyverno.io(the vendored kyverno chart pulls fromKyverno's own registry, not ghcr.io which the imageset mirrors)
Documentation (1 commit):
documented target, Docker registry as an optional appendix, the three
known timing issues documented with one-line fixes
Cleanup (3 commits):
scripts/deprecated/README.mdwith rationale)23-live.sh)argocd-loginrestored as a diagnostic tool (retired in error)Known timing issues (documented, not fixed in this PR)
Three issues occur consistently in disconnected deployments, each with a
documented workaround in the runbook:
multicluster-enginesubscription withsource: redhat-operators(disabled in airgap). Fix: patch the subscription source post-install.
retry indefinitely with stale initdata (the
cc_init_dataannotationis immutable). Fix: delete the workload pods after the KBS is healthy.
need the org in
MIRROR_REGISTRYfor the manual-mirror substitution.Fix: set
MIRROR_REGISTRY=<host>:443/<org>(not the bare host).Testing
Two consecutive from-scratch disconnected deployments (2026-10-30/31):
oc applysteps outside the documented runbookBREAKING CHANGES
None. All changes are additive or corrective.
Notes for reviewers
pattern-ui-catalogremoval is safe: nothing in the pattern values,charts, or the patterns-operator CSV references it. The image's manifest
format (OCI index wrapping Docker v2 children) is rejected by Quay.
enable-airgap-secrets.shdefault-location fix (convention path) isa revert of an earlier change that moved it to the repo directory — the
convention location is where
gen-secretswrites and the Ansible loaderreads; editing a repo-local copy causes
bootstrap_secretsto silentlyno-op.