HDDS-16025. Allow old clients to start finalization on a new server. by errose28 · Pull Request #11074 · apache/ozone

errose28 · 2026-08-20T22:54:54Z

What changes were proposed in this pull request?

An old client's finalize upgrade command to OM should still start the finalization process. The server will need to return a different response to the old client and redirect to the new finalization flow internally. This has already been handled on SCM in HDDS-15609.

Create a common base class to handle overlapping logic of the new and old request types in OM
- Add a subclass for the new and old request types to handle their unique concerns.
- Note that this does not require compatibility handling among the OMs because we must upgrade to this version before we can do ZDU with mixed OM versions.
Previously the old finalize request type had been repurposed for internal finalization among OMs, creating a compatibility issue when older clients sent that type.
- This request type is now exclusively for old clients to trigger finalization, and the new internal OM finalize request has a dedicated request type.
Remove unused method org.apache.hadoop.hdds.scm.client.ScmClient#finalizeUpgrade left over from an earlier implementation on the feature branch where finalization started from client to SCM.
Fix an idempotency bug in OM where sending finalize to an already finalized OM would leave an orphan finalization in progress marker key in the DB.

The high line count is due to the new integration test and movement of existing code in the finalize requests. Most changes do not introduce new functionality.

What is the link to the Apache JIRA

HDDS-16025

How was this patch tested?

Unit tests for request/response types added.
Integration test added to test full cluster finalization with the old finalize and status RPCs.

sodonnel

I believe this looks good. As stated in the description much of the new line count is due to tests and lifting some existing code into a new shared base class, so the change isn't as large as it first appears.

errose28 · 2026-08-21T15:21:51Z

Unfortunately this broke something at the acceptance level that I need to figure out: https://github.com/errose28/ozone/actions/runs/32425781316/job/96617583956

(cherry picked from commit 38abf22)

errose28 · 2026-08-25T21:18:37Z

The last commit fixed an existing issue on the branch where artifacts from acceptance test runs would not be uploaded, so there were no logs to see why the test was failing. With that fixed, we can now see that the upgrade acceptance test fails with:

NOT_SUPPORTED_OPERATION Finalize rejected: the following OM peers did not confirm matching software version (expected version=ZDU (100)): om1 (unreachable: DestHost:destPort om1:9862 , LocalHost:localPort om2/10.9.0.12:0. Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]), om3 (unreachable: DestHost:destPort om3:9862 , LocalHost:localPort om2/10.9.0.12:0. Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)])

Acceptance tests are still using the old CLI. Prior to this change that means they only finalized OM by submitting the same finalize request the background service does. Now they go through the proper path, so it is the first time the peer version check RPC between OMs is tested in a secure environment. We need to send this RPC as the logged-in OM user.

sodonnel · 2026-08-26T09:53:27Z

 create_stack_dumps() {
  local c pid procname
-  for c in $(docker-compose ps | cut -f1 -d' ' | grep -e datanode -e om -e recon -e s3g -e scm | grep -v -e prometheus); do
+  for c in $(docker-compose ps | cut -f1 -d' ' | grep -e datanode -e om -e recon -e s3g -e scm | grep -v -e prometheus -e s3g-haproxy); do


Did this fix it because the script was crashing due to trying to take a java thread dump from a ha-proxy process and so adding this exclusion stopped the crash and hence allowed the bundle to be captured?

Yes. I didn't document it here since it was just to collect logs for this change. I'm planning to revert it and commit it in #11039 where it belongs. Once this CI is green on my fork we will need another run on the PR anyways, so I'll revert it then.

dombizita

Thanks @errose28, one small nit, otherwise it looks good to me!

dombizita · 2026-08-26T12:06:45Z

+    verify(scmContainerLocationProtocol).finalizeUpgrade();
+  }
+
+  // TODO check if this is desired behavior


nit: I believe this is the desired behavior, right? This comment can be removed then.

Yeah fixed in the next commit.

…ng all old commands

* HDDS-14496-zdu: (200 commits) HDDS-16190. Do not use unified version framework in OM until ZDU is finalized. (apache#11025) HDDS-16154. Datanode should assign DatanodeDetails#currentVersion based on its apparent version (apache#11006) HDDS-16153. DatanodeDetails should use HDDSVersion instead of int (apache#10994) HDDS-15549. Recon's INITIAL_VERSION should not have an upgrade action (apache#10935) HDDS-15422. Stream read seek should not close stream (apache#10415) HDDS-16089. Improve MPU table evaluation efficiency in KeyLifeCycleService (apache#10949) HDDS-16169. Add old updateContainerState to handle upgrade path. (apache#11012) HDDS-16158. Fix sendDeleteKeysRequestAndClearList Authorization and FSO Trash Handling (apache#10996) HDDS-15911. Recon AI Assistant: add conversation memory for follow-up questions. (apache#10808) HDDS-15356. Make multi-buffer chunk checksum allocation-free (apache#10350) HDDS-15895. Make ServerNotLeaderException suggested leader flow IPv6-safe (apache#10807) HDDS-16118. Release snapshot DB handles before KeyDeletingService submits OM requests (apache#10987) HDDS-16125. Copy bucket snapshot in quota repair response to fix cache aliasing (apache#11004) HDDS-8082. Check for unnecessary newlines in config defaults (apache#10840) HDDS-15826. Recon: add cycle guard to NSSummary /du tree walks (apache#10723) HDDS-11620. Log SCM finalization completion (apache#10963) HDDS-16117. Failed FSO multipart complete leaks bucket namespace quota (apache#10984) HDDS-15961. Resolve linked bucket source properties consistently (apache#10859) HDDS-16072. Implement GetObjectAttributes: ETag, ObjectSize, StorageClass, and ObjectParts count (apache#10930). HDDS-16114. Avoid misleading Ozone mount warning in acceptance test results (apache#10979) ... Conflicts: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto

errose28 · 2026-08-26T18:50:03Z

Updates:

Resolve merge conflicts now that feature branch is updated with master
Deprecate old CLI classes and hide their commands
Deprecate all old RPC methods in the translators. This had only been done in some places previously.
Remove an unnecessary wait from the old client integration test
Don't use the old OzoneManager#finalizeUpgrade(clientID) method within the OM server, only use the new OzoneManager#finalizeUpgrade()
Rename TestOMFinalizeUpgradeRequestBase -> TestOMStartFinalizeUpgradeRequestBase to match its subclasses.

Once I verify acceptance tests are passing on my fork I will revert the change to testlib.sh so it can be committed in #11039

This reverts commit 5830395.

errose28 · 2026-08-26T22:39:57Z

Finally green on my fork.

sodonnel

Good work getting this all through and tracking down all the problems. LGTM

* HDDS-14496-zdu: (201 commits) HDDS-16025. Allow old clients to start finalization on a new server. (apache#11074) HDDS-16190. Do not use unified version framework in OM until ZDU is finalized. (apache#11025) HDDS-16154. Datanode should assign DatanodeDetails#currentVersion based on its apparent version (apache#11006) HDDS-16153. DatanodeDetails should use HDDSVersion instead of int (apache#10994) HDDS-15549. Recon's INITIAL_VERSION should not have an upgrade action (apache#10935) HDDS-15422. Stream read seek should not close stream (apache#10415) HDDS-16089. Improve MPU table evaluation efficiency in KeyLifeCycleService (apache#10949) HDDS-16169. Add old updateContainerState to handle upgrade path. (apache#11012) HDDS-16158. Fix sendDeleteKeysRequestAndClearList Authorization and FSO Trash Handling (apache#10996) HDDS-15911. Recon AI Assistant: add conversation memory for follow-up questions. (apache#10808) HDDS-15356. Make multi-buffer chunk checksum allocation-free (apache#10350) HDDS-15895. Make ServerNotLeaderException suggested leader flow IPv6-safe (apache#10807) HDDS-16118. Release snapshot DB handles before KeyDeletingService submits OM requests (apache#10987) HDDS-16125. Copy bucket snapshot in quota repair response to fix cache aliasing (apache#11004) HDDS-8082. Check for unnecessary newlines in config defaults (apache#10840) HDDS-15826. Recon: add cycle guard to NSSummary /du tree walks (apache#10723) HDDS-11620. Log SCM finalization completion (apache#10963) HDDS-16117. Failed FSO multipart complete leaks bucket namespace quota (apache#10984) HDDS-15961. Resolve linked bucket source properties consistently (apache#10859) HDDS-16072. Implement GetObjectAttributes: ETag, ObjectSize, StorageClass, and ObjectParts count (apache#10930). ...

errose28 added 3 commits August 20, 2026 13:45

Add compat handling with unit tests

46633a4

Add integration test for old RPCs

c4e610a

Remove unused SCM finalize method

6ca28f2

errose28 added the zdu Pull requests for Zero Downtime Upgrade (ZDU) https://issues.apache.org/jira/browse/HDDS-14496 label Aug 20, 2026

errose28 requested review from dombizita and sodonnel August 20, 2026 22:55

sodonnel approved these changes Aug 21, 2026

View reviewed changes

Fix exclusions

5830395

(cherry picked from commit 38abf22)

OM peer version check should run as OM user

fd225c6

sodonnel reviewed Aug 26, 2026

View reviewed changes

sodonnel approved these changes Aug 26, 2026

View reviewed changes

dombizita approved these changes Aug 26, 2026

View reviewed changes

errose28 added 3 commits August 26, 2026 14:06

Updates after reviewing diff, including properly deprecating and hidi…

e217dd2

…ng all old commands

Checkstyle

46436e4

errose28 mentioned this pull request Aug 26, 2026

HDDS-16020. Configure upgrade acceptance tests to run on master #11039

Open

errose28 added 3 commits August 26, 2026 16:25

Fix existing finalization idempotency bug

a713788

SCM had the same internal auth issue

1e2778c

Revert "Fix exclusions"

765ccb7

This reverts commit 5830395.

errose28 marked this pull request as ready for review August 26, 2026 22:40

sodonnel approved these changes Aug 27, 2026

View reviewed changes

dombizita approved these changes Aug 27, 2026

View reviewed changes

errose28 merged commit c46d5ed into apache:HDDS-14496-zdu Aug 27, 2026
28 of 34 checks passed

Navigation Menu

Uh oh!

HDDS-16025. Allow old clients to start finalization on a new server. - #11074

HDDS-16025. Allow old clients to start finalization on a new server.#11074
errose28 merged 11 commits into
apache:HDDS-14496-zdufrom
errose28:worktree/om-finalize-with-old-clis

errose28 commented Aug 20, 2026 •

edited

Loading

Uh oh!

sodonnel left a comment

Uh oh!

errose28 commented Aug 21, 2026

Uh oh!

errose28 commented Aug 25, 2026

Uh oh!

sodonnel Aug 26, 2026

Uh oh!

errose28 Aug 26, 2026

Uh oh!

dombizita left a comment

Uh oh!

dombizita Aug 26, 2026

Uh oh!

errose28 Aug 26, 2026

Uh oh!

errose28 commented Aug 26, 2026

Uh oh!

errose28 commented Aug 26, 2026

Uh oh!

sodonnel left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

errose28 commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

sodonnel left a comment

Choose a reason for hiding this comment

Uh oh!

errose28 commented Aug 21, 2026

Uh oh!

errose28 commented Aug 25, 2026

Uh oh!

sodonnel Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

dombizita left a comment

Choose a reason for hiding this comment

Uh oh!

dombizita Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 commented Aug 26, 2026

Uh oh!

errose28 commented Aug 26, 2026

Uh oh!

sodonnel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

errose28 commented Aug 20, 2026 •

edited

Loading