Allow scaling a Kubernetes cluster that is in Alert by bhouse-nexthop · Pull Request #14604 · apache/cloudstack

bhouse-nexthop · 2026-11-07T00:13:29Z

Description

A Kubernetes cluster in Alert can now change its size, and have worker nodes removed with nodeids. Previously scaleKubernetesCluster refused any request in Alert. So one stopped node VM blocked scaling, and blocked the CloudStack cluster-autoscaler, which scales through the same API. It also blocked removing that stopped node.

Fixes #14603

What changes

State check (KubernetesClusterManagerImpl.validateKubernetesClusterStateForScale, moved out of validateKubernetesClusterScaleParameters). In Alert it accepts a size change, or removing worker nodes by nodeids. It refuses:
- a service offering or autoscaling change, since both act on every node, including the one that is down. Sending the offerings and autoscaling setting the cluster already has is accepted;
- a request that changes neither the size nor the nodes. Such a request would do nothing and then move the cluster from Alert to Running;
- removing control, etcd or external nodes by nodeids. Scaling never acts on external nodes; they are removed with removeVirtualMachinesFromKubernetesCluster;
- any request while the first control node VM isn't running. Scaling reaches the cluster over SSH to that node;
- any request while the cluster has more or fewer VMs than nodes, for example VMs left by a failed scale-up, which a scale-up would only add to.
- any request while a control or etcd node VM is down. etcd may then have lost quorum, and new nodes can't join, so the control plane has to be recovered first;
- any request while none of the cluster's worker VMs is down (see "Down" below), leaving out external nodes, which scaling doesn't act on. The cluster is then in Alert for another reason that scaling won't fix, such as a failed start or upgrade, or a VM on a disconnected host.
What the worker gets in Alert:
- No offerings, so it only changes the size, and can't apply the cluster offering to a node type whose VMs use another one.
- The cluster record that was validated is passed on as is, rather than read again.
- The worker fails the request if the cluster's state has changed since it was validated, for example to Alert, or because another operation has started.
State machine (KubernetesCluster): Alert -> Scaling on ScaleUpRequested and ScaleDownRequested. Scaling -> Running on success and Scaling -> Alert on failure are unchanged, so the status scanner puts the cluster back in Alert if a node is still down.
Scale-up (KubernetesClusterScaleWorker):
- After provisioning, the worker waits for the nodes whose VMs aren't down to be Ready. Before, it waited for every node, and a stopped node, which is never Ready, made every scale-up time out.
- The new nodes must all be Ready. The expected count is read again on every check, in case a VM is started or stopped meanwhile. This goes through a new KubernetesClusterUtil.validateKubernetesClusterReadyNodesCount overload; the existing signature behaves as before.
- Once the count matches, the worker also waits for each new node to be Ready by name (isKubernetesClusterNodeReady, as the upgrade worker does), within the same scale timeout, before it detaches the binaries ISO. Kubernetes keeps reporting the node of a VM that was just stopped as Ready for 40-50 s (node-monitor-grace-period), and in that window it can make up the count before a new node has joined. The ISO was then detached and the new node could never join.
- The capacity check plans only the new VMs in Alert, as it does in Running. Before, it planned capacity for the whole cluster again.
Scale-down by size: workers whose VMs are down are removed first. Otherwise the order is unchanged.
Removing a node whose VM is down: the worker no longer runs kubectl drain, and deletes the Kubernetes node object directly.
- The drain waits for pod deletions that a NotReady node can never confirm, so it failed, and the scale-down with it.
- A down VM runs no pods. Once its node object is deleted, Kubernetes garbage-collects the pods and their controllers recreate them elsewhere.
"Down" means Stopped, Shutdown, Error, Destroyed or Expunging. Migrating, Unknown (a disconnected host) and the other states may still be running the VM, so those nodes are treated as before.
UI: the Scale action is shown for a cluster in Alert that doesn't autoscale. The dialog then shows only the size field, and sends only the size. An autoscaling cluster in Alert is scaled by the cluster-autoscaler itself. The per-node delete button in the cluster's instances tab is unchanged, so removing a specific node in Alert is done through the API (nodeids) for now.

Running, Created and Stopped clusters behave as before. The exception is a cluster that is still Running while one of its VMs is already down, before the status scanner has moved it to Alert. There, the down node is now removed first, isn't drained, and isn't expected to become Ready.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

Unit tests, all in plugins/integrations/kubernetes-service (mvn -pl plugins/integrations/kubernetes-service -am test -Dtest='KubernetesCluster*Test'), pass. Checkstyle on api and the plugin is clean.
New tests in KubernetesClusterManagerImplTest:
- accepted in Alert: a size change; nodeids for a worker; a size change sent with the cluster's current offerings and autoscaling setting;
- refused in Alert: a cluster or node-type offering change; an autoscaling change, including a different minsize/maxsize; a request with no size or node change; a request with the current size; removing a control node; the first control node VM stopped; more VMs than nodes; no worker VM down (all running, or only Unknown); an etcd VM down; only an external node down;
- an offering change is still accepted in Running, and Upgrading is still refused.
New tests in KubernetesClusterScaleWorkerTest: a stopped worker is picked first for a scale-down by size; the down-node count leaves out etcd nodes and newly provisioned VMs; which VM states count as down; after a scale-up, the ready-node count leaves out down nodes, each new node is then waited for by its lower-case name, and the scale-up fails if the count matches while a new node isn't Ready.
Live test: scale-up, scale-down and nodeids removal were run against a real CKS cluster in Alert (see the lab report in the comments). The first run found a scale-up reporting success before the new node joined, which the second commit fixes. The UI change was not tested in a browser.

How did you try to break this feature and the system with this change?

Other states: for Created, Running and Stopped clusters, the state check, the offering map and the existing ready-node check are unchanged. The only difference is the down-VM handling for a Running cluster with a down VM, as described above.
Restarts during a scale-up: the expected Ready count is read again on every check. A stopped VM that HA or a user starts again is then expected to rejoin, and the scale-up can't report success before the new nodes are Ready.
Hosts disconnected or VMs migrating: Unknown and Migrating don't count as down, so those nodes are still drained, and still expected to be Ready.
Concurrent requests: a second scale request is refused at validation once the first has moved the cluster to Scaling. If both are validated before that, the second now fails when its worker starts, because the state has changed. Before, it ran alongside the first. The check and the transition are still separate steps, so a very narrow window remains.
Not covered here:
- Removing control nodes by nodeids in Running records the wrong worker count. That is unchanged by this PR, and refused in Alert.
- removeKubernetesClusterNode increments its retry counter twice per pass, so it makes fewer attempts than intended.
- A request that sends only the cluster offering in Running still re-applies it to node types whose VMs use another offering.
- A worker count change without a new offering records the cluster's cores and memory using the cluster offering, not the worker offering, which new worker VMs get. A size change in Alert is always such a change, as is a size-only request in any state (the cluster-autoscaler's). Where the two offerings differ, the recorded capacity is wrong. This is a known follow-up.
- A Running VM whose node never joined, or is NotReady, is still drained, and the drain fails. Scaling up a cluster with such a node still times out, as it does in Running.
These are existing issues and are left for separate changes.

When any node VM of a running cluster isn't Running, the status scanner moves the cluster to Alert, and scaleKubernetesCluster then refuses every request: Kubernetes cluster <name> is in Alert state and can not be scaled The cluster-autoscaler scales through the same API, so one stopped node blocks adding capacity, autoscaling, and removing that node with nodeids, which is when scaling is most needed. Allow a size change, or removing nodes by nodeids, in Alert: - state machine: Alert -> Scaling on ScaleUpRequested and ScaleDownRequested. Scaling -> Running / -> Alert are unchanged, so the scanner puts the cluster back in Alert if a node is still down. - the state check moves to validateKubernetesClusterStateForScale. In Alert it refuses an offering or autoscaling change, which act on every node (a different minsize/maxsize counts, as in the worker); a request that changes neither the size nor the nodes, which would only move the cluster to Running; removing control, etcd or external nodes; any request while the first control node VM, which scaling reaches over SSH, isn't running; any request while the cluster has more or fewer VMs than nodes, for example VMs left by a failed scale-up, which a scale-up would only add to; any request while a control or etcd VM is down, as etcd may have lost quorum and new nodes can't join; and any request while none of its (non-external) worker VMs is down, as the cluster is then in Alert for another reason, such as a failed start or upgrade or a VM on a disconnected host, that scaling won't fix. Sending the offerings and autoscaling setting the cluster already has is allowed. - in Alert the worker gets no offerings, so it only changes the size and can't apply the cluster offering to another node type. The request is carried out against the cluster as validated, not a fresh read the scanner may have changed, and fails if the cluster's state has changed since it was validated (to Alert, or by another operation starting). - scale-up waits for the nodes whose VMs aren't down to be Ready, rather than every node: a stopped node is never Ready, so the scale-up always timed out. The new nodes must all be Ready, and the count is re-read on every check, in case a VM is started or stopped meanwhile. Capacity is planned for the new VMs only, as in Running. - scale-down by size removes workers whose VMs are down first. - a node whose VM is down is not drained before it is deleted. The drain waits for pod deletions that a NotReady node can never confirm, so removing the stopped node always failed. - "down" is Stopped, Shutdown, Error, Destroyed or Expunging. Migrating, Unknown and other states may still be running the VM and are treated as before. - the UI shows Scale for a cluster in Alert that doesn't autoscale, with only the size field. Signed-off-by: Brad House <bhouse@nexthop.ai>

codecov · 2026-11-07T00:19:39Z

Codecov Report

❌ Patch coverage is 63.77953% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.32%. Comparing base (d02adcd) to head (372f1fa).

Files with missing lines	Patch %	Lines
...er/actionworkers/KubernetesClusterScaleWorker.java	61.53%	18 Missing and 2 partials ⚠️
...bernetes/cluster/KubernetesClusterManagerImpl.java	75.80%	7 Missing and 8 partials ⚠️
...ubernetes/cluster/utils/KubernetesClusterUtil.java	0.00%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               4.22   #14604    +/-   ##
==========================================
  Coverage     18.32%   18.32%            
- Complexity    16644    16679    +35     
==========================================
  Files          5939     5939            
  Lines        537768   537878   +110     
  Branches      65929    65963    +34     
==========================================
+ Hits          98551    98586    +35     
- Misses       428008   428082    +74     
- Partials      11209    11210     +1

Flag	Coverage Δ
uitests	`4.08% <ø> (-0.01%)`	⬇️
unittests	`19.42% <63.77%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

A scale-up waited until the number of Ready nodes matched the provisioned nodes minus those whose VMs are down. Kubernetes keeps reporting the node of a VM that was just stopped as Ready until node-monitor-grace-period runs out (40-50 s). In that window the stopped node makes up the count before a new node has joined, so the scale-up reported success and detached the binaries ISO while the new node was still waiting for it. That node could then never join, and the next scale-up waited on it until the scale timeout. After the count matches, also wait for each new node to be Ready by name, as the upgrade worker does, within the same scale timeout. The ISO is detached only after that. If a new node isn't Ready by the timeout, the scale-up fails as before and the cluster goes to Alert. Signed-off-by: Brad House <bhouse@nexthop.ai>

bhouse-nexthop · 2026-11-09T18:48:36Z

Test report: scaling a Kubernetes cluster that is in Alert, on a live cluster

Summary: with the second commit ("Wait for each new node by name before a scale-up detaches the ISO"), every scenario passes. That includes the case that failed in my first run: the ready-node count is met by a stopped node that Kubernetes still reports as Ready. The scale-up now holds the binaries ISO until the new node is Ready by name, and then succeeds. Scale-down, nodeids removal and the refusals behave as in the first run. A normal scale-up afterwards completes without waiting on anything.

Environment

A CloudStack 24.0 (main) build with this PR, both commits, alongside some of my other open PRs. Among them are Stop concurrent CKS operations clobbering each other, recover stuck states #14609 and CKS: fix node drain on scale-down and upgrade, make a failed upgrade resumable #14611, which touch the same workers.
The PR targets 4.22; it wasn't run on a 4.22 build. The code it changes is the same on both branches: the scale worker, KubernetesClusterUtil and the scale validation in KubernetesClusterManagerImpl. That includes isKubernetesClusterNodeReady, which the fix reuses from the upgrade worker.
Three KVM hosts, Ceph RBD primary storage, one management server.
CKS cluster A: 3 control nodes and 2 workers, Kubernetes v1.33.1, isolated network.
Scale timeout at its default (3600 s). cloud.kubernetes.cluster.node.drain.timeout=60 during the run, put back afterwards.
Kubernetes marked a stopped worker's node Unknown about 50 s after its VM stopped.

Results

Scenario	Result
S1: stop a worker VM; the scanner moves the cluster to Alert	pass (Alert after 0-24 s, over 6 runs)
S2: `size` 2 -> 3 in Alert, one worker stopped, scale started as soon as the cluster is in Alert	pass, 4 runs. The ISO was detached 7-15 s after the new node was Ready. These runs didn't reach the stale-Ready window (see below)
S2, inside the stale-Ready window: the count is met by a stopped node that is still Ready	pass. The count matched 45 s after the stop. The worker then waited 1 min 50 s for the new node by name, and detached the ISO 15 s after it was Ready. The scale-up succeeded
Follow-up: a normal scale-up in Running after S2	pass, 133 s (twice)
S3: `size` 3 -> 2 in Alert: the stopped worker goes first, and isn't drained	pass (0 drain lines, node object removed, back to Running). Also `size` 3 -> 1 with two stopped workers: both removed, no drain
S4: stop the other worker, then remove it with `nodeids` in Alert	pass (0 drain lines, removed, back to Running)
S5a-e: refusals in Alert (offering, autoscaling, same size, a control node in `nodeids`, a control VM down)	pass, same messages as the first run
S5: start the stopped worker; the scanner moves the cluster back to Running	pass (Running after 21 s, 5 nodes Ready)

S2: the first run's failure, and what changed

First run (first commit only). I stopped a worker, and the cluster was in Alert 6 s later. I then scaled to size=3.
- The scale worker's first ready-node check came 43 s after the stop. Kubernetes still reported the stopped node as Ready at that point.
- So the 4 running nodes plus the stopped one made the expected 5 before the new node had booted.
- The scale-up reported success and detached the ISO 0.6 s after attaching it. The new node never joined, and the next scale-up waited on it until I attached the ISO by hand.
The change. Once the count matches, the worker also waits for each new node to be Ready by name, within the same scale timeout, before it detaches the ISO. The name is the lower-cased hostname, checked with isKubernetesClusterNodeReady.
Reaching the window again. I first repeated S2 exactly as before, starting the scale the moment the cluster went to Alert (polled every second).
- In these runs the cluster reached Alert 17-24 s after the stop, and provisioning the new VM took about 37 s.
- So the first count ran 58-63 s after the stop, after Kubernetes had already marked the node Unknown. The count never matched early, and these runs didn't reach the window.
- They do show the normal path on the new build. The ISO stayed attached until the new node was Ready, and was detached 7-15 s after.

Deterministic variant.

With the cluster already in Alert (worker 1 stopped, its node Unknown), I stopped worker 2 and immediately scaled to size=3. Worker 2 stays Ready in Kubernetes for about 50 s, while its VM is down.
The expected count is 6 provisioned nodes minus 2 down = 4. The 3 control nodes plus the stale worker 2 make 4 before the new node exists. That is the same early match as in the first run.
Timeline (worker 2 stopped at 18:39:31; node samples every 2-3 s):

18:39:39       scaleKubernetesCluster size=3 sent; cluster in Alert
18:40:15,874   INFO  KubernetesClusterScaleWorker  Attached binaries ISO for VM: <new worker> in cluster: cluster A
18:40:15,877   DEBUG KubernetesClusterUtil  Checking ready nodes for the Kubernetes cluster cluster A with total 6 provisioned nodes, 4 expected to be ready
18:40:16,334   INFO  KubernetesClusterUtil  Kubernetes cluster cluster A has 4 ready nodes now
               (k8s at 18:40:14: worker 1 Unknown, worker 2 True, new worker not registered)   <- the early match
18:40:16,686   DEBUG KubernetesClusterUtil  Failed to retrieve status for node: <new worker> in Kubernetes cluster: cluster A. Output:
               ... the same line every 15 s, 8 times in all ...
18:42:04,902   DEBUG KubernetesClusterUtil  Failed to retrieve status for node: <new worker> ...
18:42:06       new worker Ready (lastTransitionTime)
18:42:21,365   INFO  KubernetesClusterScaleWorker  Detached Kubernetes binaries from VM: <new worker> in the Kubernetes cluster: cluster A
18:42:22       API returns success (163 s)

Before the fix, the worker would have detached the ISO at 18:40:16, as it did in the first run.
Node and ISO samples around the join:

18:40:19  worker 1=Unknown worker 2=True                      | ISO on: new worker
18:40:27  worker 1=Unknown worker 2=Unknown                   | ISO on: new worker
18:42:04  worker 1=Unknown worker 2=Unknown new worker=False  | ISO on: new worker
18:42:16  worker 1=Unknown worker 2=Unknown new worker=True   | ISO on: new worker
18:42:28  worker 1=Unknown worker 2=Unknown new worker=True   | ISO on: (none)

kubectl get nodes when the API returned:

control-1   Ready      control-plane   24h     v1.33.1
control-2   Ready      control-plane   24h     v1.33.1
control-3   Ready      control-plane   24h     v1.33.1
worker-1    NotReady   <none>          21m     v1.33.1
worker-2    NotReady   <none>          5m37s   v1.33.1
worker-3    Ready      <none>          23s     v1.33.1    <- new

The cluster stayed in Alert, as both stopped workers were still in it.
size=1 then removed both stopped workers in 28 s, with no drain, and the cluster went back to Running.
A normal size=2 scale-up in Running then took 133 s. The ISO was detached 2.6 s after the count matched, with the new node already Ready.

S3, S4, S5 on the new build

S3: size=2 with one worker stopped removed that worker first, in 19 s. There were 0 drain lines for it, its node object was deleted, and the cluster was Running.
S4: nodeids=<stopped worker> in Alert removed it in 19 s, with 0 drain lines. The cluster was Running, and a size=2 scale-up afterwards took 133 s. In the first run, that scale-up waited on the stranded node.

S5a-e were all refused, with the messages from the first run:

a, b) "... is in Alert state; only its size can be changed or nodes removed. Recover the cluster before changing its service offering or autoscaling"
c)    "... is in Alert state; only its size can be changed or nodes removed, and the request does neither"
d)    "... is in Alert state; only worker nodes can be removed from it"
e)    "... is in Alert state and a control or etcd node VM is down; recover the control plane before scaling the cluster"

Not caused by this PR

My first attempt on the new build failed at once with an NPE, Cannot invoke "VirtualMachineTemplate.getUniqueName()" because "template" is null, in createKubernetesNode.
The cluster was created before 4.21. Its per-node-type template columns, added by CKS Enhancements #9102, are still NULL after the upgrade.
I've written that up separately as a stock bug. For this run I set the cluster's control and worker template ids to its template, as a cluster created on 4.21+ would have them, and put them back afterwards.

Result: pass. The fix closes the failure from my first run. A scale-up whose ready count is met early by a stale node now waits for the new node by name, keeps the ISO attached until the node has joined, and succeeds. No node is left stranded for the next scale-up. Everything else in the PR behaves as in the first run.

Not covered

The timeout path of the by-name wait: a new node that never becomes Ready, so the scale-up fails and the cluster goes to Alert. This is covered by the unit test only.
The UI's Scale form for a cluster in Alert.
A VM in Unknown (host disconnected), which the PR treats as not down.
The cluster-autoscaler driving the API.

boring-cyborg Bot added component:api component:kubernetes component:UI labels Nov 7, 2026

Navigation Menu

Repository navigation

Allow scaling a Kubernetes cluster that is in Alert - #14604

Allow scaling a Kubernetes cluster that is in Alert#14604
bhouse-nexthop wants to merge 2 commits into
apache:4.22from
bhouse-nexthop:cks-scale-in-alert

bhouse-nexthop commented Nov 7, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Nov 7, 2026 •

edited

Loading

Uh oh!

bhouse-nexthop commented Nov 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bhouse-nexthop commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What changes

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov Bot commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bhouse-nexthop commented Nov 9, 2026

Test report: scaling a Kubernetes cluster that is in Alert, on a live cluster

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bhouse-nexthop commented Nov 7, 2026 •

edited

Loading

codecov Bot commented Nov 7, 2026 •

edited

Loading