Stop concurrent CKS operations clobbering each other, recover stuck states by bhouse-nexthop · Pull Request #14609 · apache/cloudstack

bhouse-nexthop · 2026-11-07T11:44:42Z

Description

Two related fixes to how a CKS cluster's state is kept.

Operations on the same cluster can no longer run at the same time and clobber each other's state.
A cluster left in an operation's state by a management server that stopped is recovered, instead of staying stuck until the database is edited.

Concurrent operations

KubernetesClusterDaoImpl.updateState wrote the state by id alone, so two operations on one cluster both got past their first transition. The workers also ignored a failed entry transition, and carried on.

Compare-and-set (KubernetesClusterDaoImpl.updateState): the update now has WHERE id = ? AND state = ? and reports whether a row changed, as VolumeDaoImpl and VMInstanceDaoImpl already do.
Entering an operation (KubernetesClusterActionWorker.transitToOperationState): each worker moves the cluster into its operation's state before it changes anything. If another operation got there first, the worker fails with a "please retry" message. It does not move the cluster, and does not undo anything the other operation may need, such as the template launch permission.
- From a state with no transition for the event, the operation is refused. Before, it went ahead without a state change.
- Starting a cluster in Alert now takes it to Starting (new Alert -StartRequested-> Starting transition), like any other start. Before, the start ran without a state, so adding or removing nodes, a stop, or the scanner's Alert reconciliation could take the cluster partway through.
- A few operations keep going ahead without a state change, where they did that on purpose:
  - start from Error, so that a cluster whose create failed stays in Error if the start fails too, rather than going to Alert, where the reconciliation could mark it Running without the provider and CSI driver;
  - stop from Created or Error;
  - destroy from Destroying, which is a retried destroy, also used by the scanner and GC.
Holding the state: once a worker has started its operation, it only moves the cluster while the cluster is in a state the operation moved it to (KubernetesClusterActionWorker.stateTransitTo). So a failure or completion never ends another operation that has taken the cluster since. An operation that went ahead without a state change leaves the state alone when it ends, except a retried destroy, which completes Destroying.
Scaling (KubernetesClusterResourceModifierActionWorker.enterScalingState): an operation enters Scaling or ScalingStoppedCluster once, at its first scaling step. A scale or autoscale of a Created cluster still only changes the stored settings, decided from the state the worker started from.
- A scale of a Created cluster fails if a start has taken the cluster since the request was accepted. Created has no scaling transition to compare and set, so a narrow window remains there.
- A scaling only fails or completes the state when it was the one that moved the cluster. Before, a validation failure in one scale request could move a concurrent scale's Scaling to Alert.
- A failure in ScalingStoppedCluster now goes to Alert, as the state machine defines. Before, it was left in ScalingStoppedCluster.
- A failed autoscaler deployment now fails the Scaling state. Before, the cluster was left in Scaling.
Adding nodes: the cluster moves to Importing before the binaries ISO is attached, not after, so an add that is refused leaves nothing attached. On failure, the cluster is only failed if this add moved it to Importing and it is still there. Before, a failure ended whatever operation held the cluster.
Status scanner:
- The Alert-to-Stopped reconciliation now moves the cluster as it was listed. It no longer stops a cluster a user has started since.
- reconcileAlertCluster only completes a recovery that it started itself.

Stuck states

Before this change, if a management server stopped during an operation, the async job was failed when the server came back or a peer saw it leave. The cluster stayed in Scaling, Upgrading, Starting and so on, and every later operation was refused.

Recovery (KubernetesClusterManagerImpl.recoverOrphanedOperationStates, run by the status scanner on every pass): it covers a cluster in Starting, Stopping, Scaling, ScalingStoppedCluster, Upgrading, Importing, RemovingNodes or Recovering that has no IN_PROGRESS job on any management server. The scanner fails its operation; Upgrading, for example, goes to Alert. The existing Alert reconciliation then returns a healthy cluster to Running.
- A Starting cluster goes to Alert. A create that fails while waiting for kubeconfig or the dashboard already does the same.
- The cluster is read just before its job is checked, and the update is a compare-and-set on the state that was read. An operation that ends in between moves the cluster on, so the update misses.
Finding the jobs: the async jobs of start, stop, scale and upgrade now record the cluster they act on (getApiResourceId). Create, delete and add/remove nodes already did. This also fills in the resource on those commands' events.
Mixed versions during an upgrade: a management server of an older version does not record the cluster on these jobs, but the job's parameters name the cluster by its UUID. The scanner leaves alone any cluster named by such a job that is in progress. It leaves every cluster alone while such a job has no parameters.
Replaced code: the scanner's first-run handling of Starting clusters. For running VMs it tried FaultsDetected, which has no transition from Starting.

No schema or API change. The binaries ISO that an interrupted operation attached is left on the VMs. The next operation's attach replaces it.

Relation to #14604

#14604 lets a cluster in Alert be scaled. It adds Alert -> Scaling transitions, which enterScalingState uses as they are. Its own check of the state when the scale worker starts still applies. This PR closes the narrow window between that check and the transition, which #14604 mentions. The two branches merge with one trivial conflict in KubernetesClusterManagerImplTest, where both add tests at the end.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

mvn -pl plugins/integrations/kubernetes-service verify passes: all of the plugin's unit tests, plus checkstyle.
New tests:
- KubernetesClusterDaoImplTest: the update is restricted to the current state, and returns false when no row changed.
- KubernetesClusterActionWorkerTest:
  - transitToOperationState moves the cluster;
  - it fails when another operation moved the cluster first;
  - it fails in a state with no transition, unless that state is one where the operation goes ahead without one;
  - failOperationIfStillIn leaves a cluster that has moved on.
- KubernetesClusterResourceModifierActionWorkerTest:
  - enterScalingState transitions once for all steps of an operation;
  - it fails when another operation got there first;
  - it leaves a never-started cluster in its state;
  - it fails for one that a start has taken since.
- KubernetesClusterManagerImplTest:
  - recovery fails only operations with no job;
  - it waits while a job for an unknown cluster is in progress;
  - it skips the job lookup when no cluster is in an operation's state;
  - it leaves a cluster whose operation ended, or that gained a job since the scan;
  - it sends a Starting cluster to Alert.
Not tested on a live cluster. I have not run concurrent operations, or restarted a management server mid-operation, against a real CKS cluster. The behaviour is covered by reading the code and by the unit tests above.

How did you try to break this feature and the system with this change?

Two operations at once: only one gets past its first transition. The other fails without touching the state. A scale that fails validation before entering Scaling no longer fails another scale's Scaling.
An operation that ends while the scanner looks at it: the scanner reads the cluster, then its job, then makes a compare-and-set on the state it read. An operation that ends in between moves the cluster on, so the update misses. To hit a live operation, one operation would have to end, and another start and reach the same state, between those reads and the update right after them.
Job cleanup: recovery trusts that a running operation has an IN_PROGRESS job. The async job GC expunges unfinished jobs older than job.expire.minutes, which is 1440 by default. If that setting is lowered below the length of a long operation, recovery can fail the operation while it is still running.
Rolling upgrade of management servers: jobs started by older servers carry no cluster id. Recovery leaves alone the clusters they name.
Behaviour that changes on purpose: add or remove nodes, upgrade, scale and autoscale used to go ahead, without a state change, from states that do not allow them. They are now refused. Examples are adding nodes to a Stopped cluster, and stopping or destroying a cluster that is in the middle of a scale or upgrade. These operations used to run alongside the operation in progress.
Not covered here, unchanged by this PR:
- A stop can still take over a Starting cluster: the state machine allows Starting -> Stopping.
- Two concurrent destroy retries, or two concurrent starts of an Error cluster, still both go ahead, as before.
- A create interrupted in its last steps, after the dashboard check, is recovered to Alert, and the Alert reconciliation can then mark it Running without the provider and CSI driver. This is the same window in which a create that fails already ends in Alert.
- An upgrade interrupted by a management server stop is recovered to Alert, and then Running, with some nodes upgraded. The cluster's version is unchanged, so the upgrade can be run again.

codecov · 2026-11-07T11:52:09Z

Codecov Report

❌ Patch coverage is 51.46199% with 83 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.34%. Comparing base (d02adcd) to head (fee9834).

Files with missing lines	Patch %	Lines
...r/actionworkers/KubernetesClusterActionWorker.java	67.39%	9 Missing and 6 partials ⚠️
...bernetes/cluster/KubernetesClusterManagerImpl.java	71.05%	9 Missing and 2 partials ⚠️
...er/actionworkers/KubernetesClusterScaleWorker.java	0.00%	11 Missing ⚠️
...bernetes/cluster/dao/KubernetesClusterDaoImpl.java	62.06%	10 Missing and 1 partial ⚠️
...KubernetesClusterResourceModifierActionWorker.java	55.00%	7 Missing and 2 partials ⚠️
...ster/actionworkers/KubernetesClusterAddWorker.java	0.00%	5 Missing ⚠️
...er/actionworkers/KubernetesClusterStartWorker.java	0.00%	4 Missing ⚠️
.../kubernetes/cluster/ScaleKubernetesClusterCmd.java	0.00%	3 Missing ⚠️
.../kubernetes/cluster/StartKubernetesClusterCmd.java	0.00%	3 Missing ⚠️
...r/kubernetes/cluster/StopKubernetesClusterCmd.java	0.00%	3 Missing ⚠️
... and 5 more

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #14609      +/-   ##
============================================
+ Coverage     18.32%   18.34%   +0.01%     
- Complexity    16644    16665      +21     
============================================
  Files          5939     5939              
  Lines        537768   537892     +124     
  Branches      65929    65950      +21     
============================================
+ Hits          98551    98676     +125     
+ Misses       428008   427995      -13     
- Partials      11209    11221      +12

Flag	Coverage Δ
uitests	`4.08% <ø> (ø)`
unittests	`19.43% <51.46%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…tates A Kubernetes cluster's state was written by id alone, so two operations started on the same cluster at once both got past their first transition. Each then ran its steps against VMs the other was changing, and whichever finished last decided the final state. The status scanner could also overwrite the state of an operation that started after it read the cluster. The state is now compared and set in one update, as the VM and volume states are. Each operation moves the cluster into its own state before it changes anything, and fails without touching the cluster when another operation got there first. From then on it only moves the cluster while the cluster is in a state the operation moved it to. Starting a cluster in Alert now moves it to Starting like any other start. A start from Error, a stop from Created or Error, and a retried destroy still go on without a state change, as they did before; the start and stop then leave the state alone when they end. From any other state with no transition for it, an operation is now refused instead of running alongside the one that holds the cluster. The scanner only completes a stop or a recovery that it started itself. A management server that stopped during an operation left the cluster in that operation's state. Nothing moved it on, and every later operation on it was refused until the database was edited by hand. The scanner now fails the operation of a cluster in an operation's state when no job is in progress for it on any management server. The existing Alert reconciliation then returns a healthy cluster to Running. To find those jobs, the start, stop, scale and upgrade jobs now record their cluster, as the create, delete and node jobs already did. A cluster named by a job in progress on a management server of an older version, which does not record it, is left alone. This replaces the scanner's first-run handling of Starting clusters, which tried a transition that does not exist. No schema or API change. Signed-off-by: Brad House <bhouse@nexthop.ai>

bhouse-nexthop · 2026-11-07T12:08:22Z

Updated the commit (now fee9834af4) after another review pass:

Workers hold their state. Once a worker has started its operation, it only moves the cluster while the cluster is in a state that operation moved it to. A failure or completion therefore no longer ends another operation that has taken the cluster since. An operation that went ahead without a state change (stop from Created or Error, start from Error) leaves the state alone when it ends. Before, a stop of an Error cluster racing a delete could move Destroying to Destroyed. A retried destroy still completes Destroying.
Start from Alert takes the cluster. New Alert -StartRequested-> Starting transition. So adding or removing nodes, a stop, or the scanner's Alert reconciliation can no longer take the cluster partway through a start. A start from Error still goes ahead without a state, so that a cluster whose create failed stays in Error if the start fails too.
A failed autoscaler deployment now fails the Scaling state. Before, the cluster was left in Scaling.
Restoring the cluster object after a missed update. KubernetesClusterDaoImpl.updateState now puts back the state the caller read when the conditional update matches no row. Before, the object reported the next state, which the database never had.
Older management servers only hold back the clusters they name. A job in progress on a management server of an older version only stops recovery of the cluster that its parameters name by UUID, not of every cluster. Recovery also lists the clusters with one query instead of one per state.

KubernetesClusterResourceModifierActionWorker keeps its own copy of the start state, rather than reusing KubernetesClusterScaleWorker.originalState, to avoid conflicting with #14604, which edits those lines.

mvn -pl api,plugins/integrations/kubernetes-service install passes (100 plugin unit tests, checkstyle). Still not tested on a live cluster.

bhouse-nexthop · 2026-11-09T17:29:58Z

Test report: concurrent CKS operations and stuck-state recovery, on a live cluster

Summary: concurrent operations are refused cleanly, and a cluster left in Upgrading or Scaling by a killed management server is recovered on the scanner's first pass. Starting a cluster in Alert holds Starting until the start ends. The recovery fixes the state only, as the description says. What the interrupted operation left behind is still there, and two of those leftovers needed a manual step. Details are below.

Environment

A CloudStack 24.0 (main) build with this PR, alongside some of my other open PRs. Among them are Allow scaling a Kubernetes cluster that is in Alert #14604 (scaling in Alert) and CKS: fix node drain on scale-down and upgrade, make a failed upgrade resumable #14611 (drain and resumable upgrade), which touch the same workers.
The PR targets 4.22; it wasn't run on a 4.22 build. The state machine, DAO and scanner code it changes is the same on both branches. One finding below only applies to main.
Three KVM hosts, Ceph RBD primary storage, one management server.
A CKS cluster with 1 control node and 1 worker, isolated network, upgraded from v1.32.5 to v1.33.1 during the test.
The management server was killed with systemctl kill -s KILL cloudstack-management and started again. Its java process restarted each time. The scanner's first pass is 300 s after start.

Results

Scenario	Result
S1a: two `size=2` scales at once	pass: one runs, the other is refused with "please retry"
S1b: a scale and a stop at once	pass: the stop is refused, and the scale completes
S2a: `addNodesToKubernetesCluster` on a Stopped cluster	pass: refused, and no ISO is attached to the candidate VM
S2b: stop while Scaling	pass: refused
S3a: management server killed during an upgrade, after the first node	pass: Upgrading -> Alert on the first scanner pass, then Running. On main the retry needs a manual ISO detach (see the main-only finding)
S3b: management server killed during a scale-up	pass for the state: Scaling -> Running on the first scanner pass. It leaves an unjoined VM and missing SSH rules (see the leftovers)
S4: start a cluster in Alert	pass: Starting is held through the whole start, then Running once

S1: concurrent operations

Two scaleKubernetesCluster size=2 requests sent together:

first:  "Kubernetes cluster cluster-b changed from state Running to Scaling before ScaleUpRequested could start; another operation may be in progress, please retry once it has finished"
second: OK -> size 2, one new worker

A scale and, 0.3 s later, a stop:

scale: OK
stop:  "Kubernetes cluster cluster-b is in state Scaling, which does not allow StopRequested"

S2: refusals from states with no transition

addNodesToKubernetesCluster with a stopped candidate VM, on a Stopped cluster:
```
"Failed to add nodes to cluster ID 1 due to: Kubernetes cluster cluster-b is in state Stopped, which does not allow AddNodeRequested"
```
The cluster stayed Stopped, and the candidate VM had no ISO attached. Before this PR, the ISO was attached before the state change.
A stop while a scale-up was running was refused with the same "does not allow StopRequested" message. The scale completed.

S3a: management server killed during an upgrade

I upgraded v1.32.5 to v1.33.1, and killed the management server once the control node reported v1.33.1. While Upgrading, a scale was refused ("is in Upgrading state and can not be scaled").

After the restart:

Alert after 296s   (the scanner's first pass)
Running after 32s  (the existing Alert reconciliation)
WARN Kubernetes cluster cluster-b was in state Upgrading with no operation in progress, as its operation ended without moving it on, for example because the management server running it stopped; marked the operation as failed

The upgrade job was marked failed.

The cluster was left as the killed job had left it: the control node on v1.33.1 and cordoned, the worker on v1.32.5, and the cluster's version still v1.32.5.
Main-only finding. Rerunning the upgrade failed at once:
```
Failed to attach binaries ISO for VM : cluster-b-control-... : InvalidParameterValueException: The specified ISO is already attached to this Instance.
```
- The killed job had left the v1.33.1 ISO on both VMs. The description says "The next operation's attach replaces it", which is true on 4.22.
- On main, TemplateManagerImpl.enforceCdromAttachLimits (added by Multiple CD-ROM / ISO Support Per VM #13101, multiple CD-ROMs per VM) refuses an ISO that is already attached.
- I've written this up separately as a main-only stock bug, with a proposed fix in attachIsoKubernetesVMs. It will matter when this PR is forward-merged.
After I detached the ISO from both VMs, the same upgrade completed in 116 s. Both nodes were upgraded, the control node again, because its version hadn't been recorded before the kill. Both were uncordoned, and the cluster was Running on v1.33.1.

S3b: management server killed during a scale-up

I started size=2 and killed the management server 20 s after the new VM appeared in the cluster.

After the restart:

WARN Kubernetes cluster cluster-b was in state Scaling with no operation in progress, ... marked the operation as failed
Scaling -> Running after 296 s (Scaling -OperationFailed-> Running is the existing transition)

Leftovers (not something the PR claims to repair):
- An unjoined VM. The new worker VM was left in the cluster, Running but never a Kubernetes node, with the size still 1. A stock scale-up that fails also leaves its VMs.
- No SSH rules. The kill came while the scale was rebuilding the cluster's network rules, which it does by revoking all port forwards and adding them again. The cluster was left with no SSH port-forward rules, and the SSH firewall rule was stuck in Add.
- The next start hung. S4's start retried kubeconfig over SSH and would have run until the 3600 s start timeout. I recreated the firewall rule and the three port forwards by hand, in VM-map order, as the scale would have, and the start then completed.
- Removing the unjoined VM. I stopped it and tried to remove it with nodeids. Allow scaling a Kubernetes cluster that is in Alert #14604 refused: "has 3 VMs for 2 nodes; recover the cluster before scaling it". I set node_count to 2 in the database, to match the 2 worker VMs. A size=1 scale in Alert then removed the stopped VM with no drain, and the cluster was Running with 1 control node and 1 worker. I also corrected the cluster's cores and memory totals, which my node_count edit had thrown off.
I suggest the stuck-state recovery could at least log what an interrupted Scaling may have left (VMs not in the node count, rules being rebuilt). Repairing it is a separate change.

S4: starting a cluster in Alert

I stopped a worker VM, and the cluster was in Alert after 21 s. I then sent startKubernetesCluster.
States sampled every second, through the S3b rule repair above:
```
Alert -> Starting (about 200 samples) -> Running (once)
```
The state didn't change during the start, and no other operation or scanner pass took the cluster midway.

Result: pass, with the main-only ISO issue and the S3b leftovers noted above.

Concurrent operations are refused with clear messages.
Operations from states with no transition are refused before they change anything.
A cluster stuck in Upgrading or Scaling after a management server stop is recovered on the scanner's next pass.

Not covered

More than one management server (a peer seeing the server leave), and the mixed-version job-parameter fallback.
Recovery from Starting, Stopping, Importing, RemovingNodes or Recovering. Only Upgrading and Scaling were interrupted.
The scale of a Created cluster racing a start.
The Alert-to-Stopped reconciliation racing a user start.

boring-cyborg Bot added the component:kubernetes label Nov 7, 2026

bhouse-nexthop mentioned this pull request Nov 7, 2026

CKS: fix node drain on scale-down and upgrade, make a failed upgrade resumable #14611

Open

3 of 12 tasks

bhouse-nexthop force-pushed the cks-stuck-state-recovery branch from f5c2ea9 to fee9834 Compare November 7, 2026 12:08

boring-cyborg Bot added the component:api label Nov 7, 2026

bhouse-nexthop mentioned this pull request Nov 8, 2026

CKS: change the node settings of a cluster and restart its nodes with them, a few at a time #14616

Draft

2 of 9 tasks

bhouse-nexthop mentioned this pull request Nov 9, 2026

Allow scaling a Kubernetes cluster that is in Alert #14604

Open

3 of 12 tasks

Navigation Menu

Repository navigation

Stop concurrent CKS operations clobbering each other, recover stuck states - #14609

Stop concurrent CKS operations clobbering each other, recover stuck states#14609
bhouse-nexthop wants to merge 1 commit into
apache:4.22from
bhouse-nexthop:cks-stuck-state-recovery

bhouse-nexthop commented Nov 7, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Nov 7, 2026 •

edited

Loading

Uh oh!

bhouse-nexthop commented Nov 7, 2026

Uh oh!

bhouse-nexthop commented Nov 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bhouse-nexthop commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Concurrent operations

Stuck states

Relation to #14604

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov Bot commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bhouse-nexthop commented Nov 7, 2026

Uh oh!

bhouse-nexthop commented Nov 9, 2026

Test report: concurrent CKS operations and stuck-state recovery, on a live cluster

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bhouse-nexthop commented Nov 7, 2026 •

edited

Loading

codecov Bot commented Nov 7, 2026 •

edited

Loading