Skip to content

fix(airgap): disconnected deployment support and bootstrap fixes - #164

Merged
butler54 merged 9 commits into
validatedpatterns:mainfrom
butler54:feat/airgap-disconnected-support
Nov 2, 2026
Merged

butler54 merged 9 commits into
validatedpatterns:mainfrom
butler54:feat/airgap-disconnected-support

Conversation

@butler54

@butler54 butler54 commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

PR: Airgap/Disconnected Deployment Support

Summary

This PR adds comprehensive disconnected (airgapped) deployment support to the
CoCo pattern, validated end-to-end on a real bare-metal TDX cluster with two
consecutive clean-slate deployments against a Quay registry as the sole
mirror target.

What's included

Imageset correctness (4 commits):

  • Align pins with installer version (4.22.17, trustee-operator 1.2.1)
  • Add defaultChannel overrides for operators whose default channels are
    filtered out by oc-mirror (gitops latest, gpu-operator v26.7)
  • Disable CNV guest images (virtio-win:latest is a hard mirror error)
  • Disable gpu-operator-certified (matches the disabled subscription)
  • Drop pattern-ui-catalog (hybrid OCI-index manifest rejected by Quay;
    nothing in the pattern or operator CSV deploys it)
  • Bump hashicorp-vault pin to match the chart's resolved image (K16
    range-pin aging: values use ~0.1.10 which resolves 0.1.11)

Airgap bootstrap (3 commits):

  • DNS-based mirror registry identity (FR-017: artifacts survive address changes)
  • Complete the airgap secrets toggle in enable-airgap-secrets.sh (handles
    tdxCollateral and intelPcsApiKey markers, not just registryCaCert and
    bootstrap_secrets)
  • enable-airgap-secrets operates at the values-secret convention location
    (~/values-secret-<pattern>.yaml — where gen-secrets writes and the
    Ansible loader reads)

Operational fixes (2 commits):

  • Trustee SNP VCEK: capture full chip_id instead of truncating to 64 hex
  • Manual ITMS covers reg.kyverno.io (the vendored kyverno chart pulls from
    Kyverno's own registry, not ghcr.io which the imageset mirrors)

Documentation (1 commit):

  • Complete runbook rewrite from 1600 lines to ~350: Quay as the primary
    documented target, Docker registry as an optional appendix, the three
    known timing issues documented with one-line fixes

Cleanup (3 commits):

  • 13 unused Makefile targets retired (bodies preserved in
    scripts/deprecated/README.md with rationale)
  • 1 unreferenced script quarantined (23-live.sh)
  • argocd-login restored as a diagnostic tool (retired in error)

Known timing issues (documented, not fixed in this PR)

Three issues occur consistently in disconnected deployments, each with a
documented workaround in the runbook:

  1. MCE subscription timing: the MCH operator creates the
    multicluster-engine subscription with source: redhat-operators
    (disabled in airgap). Fix: patch the subscription source post-install.
  2. Initdata staleness: kata-cc pods created before the KBS is ready
    retry indefinitely with stale initdata (the cc_init_data annotation
    is immutable). Fix: delete the workload pods after the KBS is healthy.
  3. ITMS org-prefix: Quay-style mirrors serving content under an org
    need the org in MIRROR_REGISTRY for the manual-mirror substitution.
    Fix: set MIRROR_REGISTRY=<host>:443/<org> (not the bare host).

Testing

Two consecutive from-scratch disconnected deployments (2026-10-30/31):

  • OCP 4.22.17 SNO on bare-metal TDX (Intel Xeon, node-02)
  • Quay registry as the sole mirror target (org-prefixed paths)
  • All 15 ArgoCD applications Synced+Healthy
  • KBS attestation returning 200 (offline TDX collateral, PCK provisioned)
  • Confidential VM workloads pulling images from the mirror inside the CVMs
  • Zero manual oc apply steps outside the documented runbook

BREAKING CHANGES

None. All changes are additive or corrective.

Notes for reviewers

  • The pattern-ui-catalog removal is safe: nothing in the pattern values,
    charts, or the patterns-operator CSV references it. The image's manifest
    format (OCI index wrapping Docker v2 children) is rejected by Quay.
  • The enable-airgap-secrets.sh default-location fix (convention path) is
    a revert of an earlier change that moved it to the repo directory — the
    convention location is where gen-secrets writes and the Ansible loader
    reads; editing a repo-local copy causes bootstrap_secrets to silently
    no-op.

butler54 and others added 7 commits October 31, 2026 07:33
…or 1.2.1

The imageset pinned OCP 4.22.8 while the deployed installer is 4.22.17,
and trustee-operator 1.2.0 while values-baremetal pins CSV v1.2.1. The
mirror must contain the exact release the installer consumes (K12) and
the operator bundles the pattern resolves.

Version audit: specs/002-multi-tranche-testing/version-audit.md (coco-gsd)
… in imageset

oc-mirror v2 fails catalog filtering when the package's catalog default
channel is filtered out: openshift-gitops-operator defaults to 'latest'
(we pin gitops-1.22) and gpu-operator-certified defaults to v26.7 (we
pin v26.3). The defaultChannel field re-points the default to the
pinned channel.

Known-failure class recorded as K13 in the T1 mirror checklist (coco-gsd
specs/002-multi-tranche-testing/mirror-known-failures.md).
…t is a hard mirror error

registry.redhat.io rejects :latest for container-native-virtualization/virtio-win
('must be pulled with an explicit version'), which fails oc-mirror on every run.
The three guest images are documented-optional (~2 GB each) and unconsumed:
Phase 26 sets enableCommonBootImageImport: false, so CNV never imports them.
Commenting them out also removes the large-blob timeout flakiness (rhel8/rhel9
guest images hit context-deadline-exceeded on upload).

T1 mirror run evidence: coco-gsd specs/002-multi-tranche-testing/t1-progress.md
…disabled subscription

The gpu-operator subscription is commented out in values-baremetal.yaml
(NVIDIA bundle mirror issues); nothing in T1 consumes the bundle. The
nvcr.io related images (dcgm, driver) hit repeated copy timeouts across
two resumable retries, blocking mirror certification for content the
disconnected deployment never pulls.

Tranche 2 re-enables the GPU operator at v26.7 (pawelpros cherry-pick),
which mandates its own re-mirror + re-certification per the T022 trigger
logic — at that point the bundle returns to the imageset.
The mirror registry is now identified by its DNS name everywhere:
- Phase 0-0: canonical MIRROR_REGISTRY=user-jump.int.apac-tech-lab.net:8443,
  with the resolved IP recorded as a generation-time fact only
- Phase 0-1b: cert CN/SAN carry the DNS name (IP SAN kept for transition)
- Phase 0-1h: certs.d trust entries for both name and IP
- oc-mirror targets the DNS name so generated cluster-resources (IDMS/ITMS/
  CatalogSources) reference it — no IP enters the ignition chain
- itms-manual-mirrors placeholders substituted from MIRROR_REGISTRY at apply
  time (was applied verbatim with the literal placeholder)
- git-server URLs and repoURL hints use the DNS name

Lab IPs are not stable (the jump host moved subnets on 2026-09-15, breaking
the T013 install through the reserved-VLAN routing); DNS is maintained
through moves and the node's ignition resolver uses it.

Spec: FR-017 (specs/002-multi-tranche-testing, R6)
…crets.sh

The script only uncommented registryCaCert and bootstrap_secrets. The
branch-pair contract for disconnected testing also requires tdxCollateral
enabled (offline DCAP collateral) and intelPcsApiKey disabled (manual PCK
provisioning via dcap-pck.py in airgap; the connected key file must not
be required by load-secrets).

- Wrap the tdxCollateral block in AIRGAP-TDX-COLLATERAL markers in
  values-secret.yaml.template (gen-secrets copies the template verbatim,
  so markers flow through to the generated file)
- Uncomment both marker blocks; comment out the intelPcsApiKey entry
- Idempotent; verified against a fresh template copy (YAML-valid output,
  airgap secrets active, intelPcsApiKey disabled)

Contract: specs/002-multi-tranche-testing/contracts/branch-pair.md (coco-gsd)
…64 hex

collect-snp-vcek-urls.sh extracted the hardware ID with
`grep -oP '[0-9a-fA-F]{64}'`, truncating the chip_id to the first 32
bytes. The SNP verifier derives the on-disk VCEK directory from the full
hex(chip_id) — 128-hex for non-Turin EPYC, 16-hex for Turin — so a
truncated value can never match and attestation fails with
POST /kbs/v0/attest returning 401, leaving workloads in
CreateContainerError.

Extract the full chip_id as the KDS URL's trailing path segment (query
stripped) without truncating.

Refs validatedpatterns/trustee-chart#45.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@butler54
butler54 requested a review from a team November 2, 2026 01:38
@butler54 butler54 changed the title Airgap/disconnected deployment support fix(airgap): disconnected deployment support — imageset alignment, bootstrap fixes, and runbook rewrite Nov 2, 2026

@butler54 butler54 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes were slimmed too far

Comment thread values-global.yaml Outdated
# Connected mode: repoURL is inferred from your local git remote.
# Disconnected: uncomment and set to the HTTP URL served by git-http-server.py
# repoURL: http://JUMP_HOST_IP:8080/coco-pattern.git
# repoURL: http://user-jump.int.apac-tech-lab.net:8080/coco-pattern.git

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an internal URL don't expose it.

Comment thread Makefile Outdated
Comment on lines +162 to +171

.PHONY: argocd-login
argocd-login: ## Extract ArgoCD credentials from cluster and log in with argocd CLI (diagnostic)
@ARGOCD_NS=$$(oc get argocd -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null) && \
ARGOCD_NAME=$$(oc get argocd -n $$ARGOCD_NS -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) && \
ARGOCD_ROUTE=$$(oc get route $${ARGOCD_NAME}-server -n $$ARGOCD_NS -o jsonpath='{.spec.host}' 2>/dev/null) && \
ARGOCD_PASS=$$(oc get secret $${ARGOCD_NAME}-cluster -n $$ARGOCD_NS -o jsonpath='{.data.admin\.password}' 2>/dev/null | base64 -d) && \
echo "ArgoCD URL: https://$$ARGOCD_ROUTE" && \
echo "Username: admin" && \
echo "Password: $$ARGOCD_PASS"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has not actually logged into argocd. Ensure it logs in.

Comment thread Makefile
ARGOCD_CLI_DIR ?= $(HOME)/.local/bin

.PHONY: argocd-install
argocd-install: ## Download argocd CLI from the cluster's ArgoCD image into ~/.local/bin

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add this utiluty back in

Comment thread Makefile
@scripts/airgap-post-install.sh --fix-manifest-lists

.PHONY: airgap-sync-repos
airgap-sync-repos: ## Push working copy changes to bare HTTP repos and restart git server

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add this back in.

Comment thread Makefile
OC_MIRROR_WORKSPACE ?= file://$(HOME)/oc-mirror-workspace

.PHONY: airgap-mirror
airgap-mirror: ## Mirror content to disconnected registry (requires MIRROR_REGISTRY)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this just a wrapper of the mirroring process? isn't it still useful?

Comment thread Makefile
@scripts/collect_firmware_refvals.py --platform baremetal

.PHONY: collect-azure-refvals
collect-azure-refvals: ## Collect PCR reference values (Azure)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is still required.

@butler54
butler54 force-pushed the feat/airgap-disconnected-support branch 3 times, most recently from 29a997c to 95a98e6 Compare November 2, 2026 02:13
@butler54 butler54 changed the title fix(airgap): disconnected deployment support — imageset alignment, bootstrap fixes, and runbook rewrite fix(airgap): disconnected deployment support and bootstrap fixes Nov 2, 2026
@butler54
butler54 force-pushed the feat/airgap-disconnected-support branch 4 times, most recently from a1d012c to a8db4eb Compare November 2, 2026 02:33
…unbook

Review feedback on the PR:
- argocd-login now actually logs in via the argocd CLI when available
  (falls back to printing credentials with a hint to install the CLI)
- Scrubbed a leaked internal URL from a commented example in
  values-global.yaml
- Added the rewritten deploy runbook (Quay primary, Docker registry
  optional appendix)
- All 31 Makefile targets retained (multi-platform: Intel TDX, AMD SNP,
  Azure)
- Archived superseded documentation in deprecated/ directories

No targets were removed — the earlier retirement was too aggressive.
@butler54
butler54 force-pushed the feat/airgap-disconnected-support branch from a8db4eb to 1b3d31a Compare November 2, 2026 02:50
@butler54
butler54 merged commit 3ec7033 into validatedpatterns:main Nov 2, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants