[bhouse-nexthop]

Description

This PR pins the cluster autoscaler and the cloud controller manager (CCM) that scripts/util/create-kubernetes-binaries-iso.sh bundles into the CKS binaries ISO, and writes a list of the ISO's contents into the ISO. Because the autoscaler now changes with the Kubernetes version, a cluster upgrade also redeploys the autoscaler from the new ISO.

Fixes #14648

Related: #14598 fixes the same download problems in the node-side scripts (deploy-provider, autoscale-kube-cluster, deploy-csi-driver). This PR doesn't touch those scripts, so the two don't conflict.

What was wrong

• Autoscaler:
  • The manifest came from kubernetes/autoscaler master, and its image is apache/cloudstack-kubernetes-autoscaler:latest.
  • That tag was last pushed in March 2022 and is a 1.24.0-alpha development build. So every ISO shipped the same old autoscaler, whatever its Kubernetes version.
  • The cluster-autoscaler only supports the Kubernetes minor version it was released with.
• CCM:
  • The manifest came from cloudstack-kubernetes-provider main, with whatever image main named on the day of the build.
  • For a long time that was v1.1.0, which does not initialise nodes (Fix providerID implementation cloudstack-kubernetes-provider#58, fixed in v1.2.0).
• No record: nothing in the ISO said which versions it held, so two ISOs for the same Kubernetes version could differ.
• Silent download failures: most downloads used curl without -f, so a 404 page was saved and bundled as a manifest or binary.

What changed

Autoscaler (ISO builder)

• By default, the script picks the newest cluster-autoscaler-<minor>.<patch> release tag for the ISO's Kubernetes minor version.
  • It finds the tag with git ls-remote, so no API token is needed.
  • Today, 1.31.x gets 1.31.5 and 1.33.x gets 1.33.6.
• AUTOSCALER_VERSION=1.33.4 picks an exact release instead.
  • The script warns if the release is for a different minor version.
  • The build stops with an error if no release exists for the minor version, for example a new Kubernetes minor before its autoscaler release.
• The manifest comes from that release tag.
  • I checked that the CloudStack example (cloudprovider/cloudstack/examples/cluster-autoscaler-standard.yaml) exists at the latest tag of every minor from 1.20 to 1.37, so no fallback manifest is needed.
  • Before 1.20 the example doesn't exist at the release tags, so builds for those (long EOL) versions now fail on the download.
• The image is replaced with the upstream image of the same release, registry.k8s.io/autoscaling/cluster-autoscaler:v<version>.
  • The example names the unversioned apache/cloudstack-kubernetes-autoscaler:latest at every tag, so it can't be used as is.
  • The upstream image is built with all cloud providers, CloudStack included.
  • It runs ./cluster-autoscaler from /, as the manifest expects.
  • It is published for amd64 and arm64.
• Extra RBAC.
  • The CloudStack example's ClusterRole hasn't kept up with newer releases. They also watch VolumeAttachments (from 1.32) and DRA objects (DRA is on by default from 1.35).
  • Without read access to those, the autoscaler never finishes syncing its caches and never scales (Failed to watch *v1.VolumeAttachment kubernetes/autoscaler#7663, ClusterRole doesn't have volumeattachments resource permissions to the Cluster Autoscaler kubernetes/autoscaler#8678). The AWS examples were fixed the same way.
  • The script appends a cluster-autoscaler-extra ClusterRole and binding with that read access. At 1.36.2 that makes the CloudStack example's permissions match the AWS example's.
  • It only adds read permissions, so it is harmless on older versions.

CCM (ISO builder)

• The default is v1.2.0, the current release and the first one that initialises nodes.
• CCM_VERSION=v1.3.0-rc1 picks another release tag.
• The manifest comes from that tag, and its image is set to apache/cloudstack-kubernetes-provider:<tag>. This is needed because a tag's manifest doesn't always name its own image: v1.3.0-rc1's names v1.2.0.

versions.txt in the ISO

• It lists the Kubernetes, CNI plugins, crictl, CCM, CSI driver, autoscaler and etcd versions.
• It also lists the URL of every file downloaded, and every bundled image with its digest.
• It is not a .yaml file, so the node scripts that copy *.yaml from the ISO ignore it.

Download and build robustness (ISO builder)

• Downloads use curl -fsSL --retry 3, one file per call, so a failed download stops the build. A multi-URL curl would only report the last file's status.
• The curl | sed pipelines for kubelet.service and 10-kubeadm.conf now download first and then edit, so a failed download is no longer hidden by the pipe.
• The CNI fallback to the old archive name is kept.
• The CSI and etcd downloads move from wget to the same helper.
• The ISO is assembled in a mktemp -d directory, removed on exit, instead of a fixed /tmp/iso. Files left behind by a failed build no longer end up in the next ISO.
• The image list is deduplicated before pulling. The CSI manifest names two images twice.
• Pulled images are removed with ctr image rm --sync.
  • Without --sync, containerd's asynchronous garbage collection could delete a layer that the next image shares, between that image's pull and its export.
  • That failed my test builds with ctr: failed to get reader: content digest ...: not found.
• The positional arguments are unchanged. Both overrides are environment variables.

Upgrade (KubernetesClusterUpgradeWorker)

• upgrade-kubernetes.sh already copies the new ISO's autoscaler.yaml to the control node, but the running autoscaler was never touched. With :latest everywhere, that didn't matter.
• Now the autoscaler's image changes with the Kubernetes version. After all nodes are upgraded and the new version is saved, a cluster with autoscaling enabled gets its autoscaler redeployed from the new ISO.
  • It runs autoscale-kube-cluster -e with the cluster's id, min and max from the database, the same command as enabling autoscaling.
  • It runs while the cluster is still Upgrading, so autoscaling can't be changed through the API in the meantime.
  • A failure is logged as a warning and leaves the previous autoscaler running. It doesn't fail the upgrade.
• The command string is now built by one helper that both callers share.

Not changed here:

• Node-side fallbacks: autoscale-kube-cluster and deploy-provider still download from master/main when the ISO doesn't have the manifests. CKS: fix the add-on scripts' download and option handling #14598 changes those same lines, so pinning them is left until it is merged.
• Kubelet unit files: kubelet.service and 10-kubeadm.conf still come from shapeblue/cloudstack-nonoss main. Their URLs are now recorded in versions.txt.
• aarch64 ISOs: images are still pulled for the build host's platform, and etcd is always the amd64 archive.
• CCM on upgrade: it is not redeployed during an upgrade, as before.
• Upgrading onto an old ISO: an upgrade to a version whose ISO was built by the old script redeploys that ISO's :latest autoscaler. That is the same autoscaler a cluster created from that ISO would get.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

ISO builds. I built ISOs with the modified script on an Ubuntu x86_64 host with containerd 2.x. I then extracted versions.txt, autoscaler.yaml, provider.yaml and the image tarballs from each ISO with isoinfo.

Build	Autoscaler (manifest tag / image)	CCM (manifest tag / image)
1.31.14, defaults	1.31.5 / registry.k8s.io/autoscaling/cluster-autoscaler:v1.31.5	v1.2.0 / apache/cloudstack-kubernetes-provider:v1.2.0
1.33.13, defaults, with etcd 3.5.21	1.33.6 / ...cluster-autoscaler:v1.33.6	v1.2.0 / ...:v1.2.0
1.33.13, AUTOSCALER_VERSION=v1.33.4 CCM_VERSION=1.3.0-rc1	1.33.4 / ...cluster-autoscaler:v1.33.4	v1.3.0-rc1 / ...:v1.3.0-rc1 (the tag's manifest named v1.2.0)

The 1.31 build was done with Calico 3.29.3 and Dashboard 2.7.0. Later builds used flannel and Headlamp manifests, because Docker Hub's anonymous pull limit stopped the Calico/Dashboard builds partway.

For each ISO I checked the following:

• autoscaler.yaml and provider.yaml name exactly those images, with imagePullPolicy: IfNotPresent.
• The <min>:<max>:<cluster-id> placeholders are intact.
• The autoscaler manifest parses into the six original objects plus the extra ClusterRole and binding.
• docker/ holds cluster-autoscaler:v<version>.tar and cloudstack-kubernetes-provider:<tag>.tar, and the image name inside each tarball matches the manifest.
• versions.txt has the right versions and URLs.
  • There is one line per image, each with a 64-hex digest.
  • The etcd version and URL appear only when etcd was requested.
• The number of image tarballs equals the number of image lines.
• No build directory is left behind.

Failure cases. Each of these stops the build with curl: (22) ... 404 and a non-zero exit:

• CCM_VERSION=v9.9.9
• AUTOSCALER_VERSION=1.33.99
• a dashboard URL that doesn't exist

Builder pieces tested on their own:

• A minor version with no autoscaler release (1.99) stops with the error asking for AUTOSCALER_VERSION.
• AUTOSCALER_VERSION=1.32.7 on 1.33 prints the mismatch warning.
• CNI v0.7.5 falls back to the old cni-plugins-amd64-<ver>.tgz name.

Unit tests. New tests in KubernetesClusterUpgradeWorkerTest cover updateAutoscaler:

• it does nothing when autoscaling is disabled or min/max is unset;
• it runs autoscale-kube-cluster -i <uuid> -e -M <max> -m <min>;
• it doesn't throw when SSH fails or returns an error.

All 82 tests in the kubernetes-service module pass, and checkstyle is clean.

Not tested:

• Deploying these ISOs on a live CKS cluster.
• In particular, running the upstream autoscaler image with the CloudStack provider against a real cluster, and a real upgrade that redeploys it.
• The RBAC addition is based on the autoscaler source and the upstream issues above.

How did you try to break this feature and the system with this change?

• Existing ISOs and clusters: unaffected at create time. The node side reads the same file names (autoscaler.yaml, provider.yaml, docker/*.tar).
• Existing build commands: the positional arguments are unchanged. Without the new variables, builds get version-matched defaults instead of master/main.
• A minor version before its autoscaler release: the build fails with a clear message instead of bundling a mismatched autoscaler. AUTOSCALER_VERSION overrides it.
• A change to an upstream manifest's layout: setting the image checks that the manifest has exactly one image: line and fails otherwise, instead of silently bundling the wrong image.
• A manifest without a trailing newline: the appended RBAC starts with a blank line, so --- can't join the last line.
• Upgrade with autoscaling disabled: nothing changes.
• Upgrade where the redeploy fails: a warning is logged and the upgrade still succeeds with the old autoscaler running.
• On an air-gapped cluster: the redeploy runs only after every node in the upgrade has imported the new ISO's images. Nodes marked for manual upgrade are the exception, but the rolling update keeps the old pod until the new one is ready.

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.32%. Comparing base (d02adcd) to head (5a49669).

Files with missing lines	Patch %	Lines
.../actionworkers/KubernetesClusterUpgradeWorker.java	87.50%	1 Missing and 1 partial ⚠️
...KubernetesClusterResourceModifierActionWorker.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               4.22   #14649   +/-   ##
=========================================
  Coverage     18.32%   18.32%           
- Complexity    16644    16646    +2     
=========================================
  Files          5939     5939           
  Lines        537768   537787   +19     
  Branches      65929    65931    +2     
=========================================
+ Hits          98551    98560    +9     
- Misses       428008   428018   +10     
  Partials      11209    11209           

Flag	Coverage Δ
uitests	4.08% <ø> (ø)
unittests	19.41% <85.71%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.